2024-03-29T10:23:41.099 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:23:41.099 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/orig.config.yaml 2024-03-29T10:23:41.132 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:23:41.214 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622640' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJuXhtiIzox6RyjMmMKLrraT+wv1+0A0OvY4n+MklX/S7b31OH0r/OKQJhnGds7aodC+WjKO8CpeIt5U8dLrpp0= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBk5LqafpU9UWpW47/yw6pdgu4pgdDLnxkVKd1NImpWtyodgt+PmSbZZ0nRc07yD+cTa1jWNYT9NCp59uKxCtw0= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:23:41.215 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:23:41.215 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:23:41.216 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:23:41.217 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:23:41.241 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:23:41.241 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:23:41.241 INFO:teuthology.packaging:ref: None 2024-03-29T10:23:41.241 INFO:teuthology.packaging:tag: None 2024-03-29T10:23:41.241 INFO:teuthology.packaging:branch: main 2024-03-29T10:23:41.241 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:23:41.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:23:41.488 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:23:41.489 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:23:41.497 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:23:41.497 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:23:41.513 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:23:41.525 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:23:41.535 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:23:41.588 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:39.369030', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJuXhtiIzox6RyjMmMKLrraT+wv1+0A0OvY4n+MklX/S7b31OH0r/OKQJhnGds7aodC+WjKO8CpeIt5U8dLrpp0='} 2024-03-29T10:23:41.613 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:39.367052', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBk5LqafpU9UWpW47/yw6pdgu4pgdDLnxkVKd1NImpWtyodgt+PmSbZZ0nRc07yD+cTa1jWNYT9NCp59uKxCtw0='} 2024-03-29T10:23:41.613 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:23:41.623 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:23:41.624 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:23:41.624 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:23:41.717 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff78c80c0d0>, signals=[15]) 2024-03-29T10:23:41.718 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:23:41.725 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:23:41.725 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:23:41.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:23:41.817 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:23:41.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:23:41.910 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:23:41.918 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-03-29T10:23:41.936 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-03-29T10:23:41.936 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-03-29T10:23:41.993 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-03-29T10:23:41.993 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-03-29T10:23:41.993 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-03-29T10:23:41.993 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:23:41.993 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-03-29T10:23:41.993 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:23:41.994 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:23:41.994 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-03-29T10:23:41.994 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:23:41.994 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:23:41.994 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:23:41.994 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:23:41.994 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:23:41.994 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:23:41.995 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-03-29T10:23:42.018 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-03-29T10:23:42.035 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-03-29T10:23:42.035 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-03-29T10:23:42.091 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-03-29T10:23:42.091 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-03-29T10:23:42.091 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-03-29T10:23:42.091 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:23:42.091 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-03-29T10:23:42.091 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:23:42.091 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:23:42.091 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-03-29T10:23:42.092 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:23:42.092 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:23:42.092 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:23:42.092 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:23:42.092 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:23:42.092 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:23:42.092 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-03-29T10:23:42.109 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:23:42.120 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:23:42.140 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:23:42.141 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:23:42.143 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:23:42.158 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:23:42.248 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:23:42.248 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:23:42.250 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:23:42.386 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:23:42.444 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:23:42.521 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:23:42.521 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:23:42.522 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:23:42.522 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:23:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:23:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:23:42.540 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-29T10:23:42.540 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-29T10:23:42.597 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:23:42.597 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-361.el9.x86_64 2024-03-29T10:23:42.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-03-29T10:23:42.598 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:23:42.598 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.14.0-361.el9.x86_64 2024-03-29T10:23:42.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-03-29T10:23:44.699 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T10:23:44.856 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:23:46.316 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-29T10:23:46.548 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-29T10:23:47.499 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-03-29T10:23:47.556 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T10:23:49.323 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-29T10:23:50.158 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-29T10:23:54.650 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.1 MB 00:00 2024-03-29T10:23:57.033 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-29T10:23:58.268 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-29T10:24:01.747 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 704 kB/s | 6.1 MB 00:08 2024-03-29T10:24:03.234 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-03-29T10:24:04.131 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-29T10:24:05.243 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-29T10:24:05.465 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:24:05.465 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:24:05.604 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:24:05.614 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:24:05.614 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-03-29T10:24:05.615 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:24:05.616 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 76 M 2024-03-29T10:24:05.616 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 123 M 2024-03-29T10:24:05.616 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:24:05.768 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T10:24:06.319 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T10:24:06.877 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T10:24:07.244 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T10:24:07.246 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:24:07.247 INFO:teuthology.orchestra.run.smithi117.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-29T10:24:07.247 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:24:07.626 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:24:07.626 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:24:08.452 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:24:08.453 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:24:10.182 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T10:24:10.732 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:24:11.330 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:24:11.419 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:12.464 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:24:12.464 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:24:12.559 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:24:12.563 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:24:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-03-29T10:24:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:24:12.565 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 76 M 2024-03-29T10:24:12.565 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 123 M 2024-03-29T10:24:12.565 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:24:12.733 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T10:24:13.425 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-29T10:24:14.033 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 23 MB/s | 34 MB 00:01 2024-03-29T10:24:14.178 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:14.313 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:14.450 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-29T10:24:14.454 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:24:14.454 INFO:teuthology.orchestra.run.smithi136.stdout:Total 40 MB/s | 76 MB 00:01 2024-03-29T10:24:14.454 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:24:14.842 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:24:14.842 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:24:15.684 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:24:15.684 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:24:17.978 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:24:18.562 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:24:18.654 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:19.015 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:19.070 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:21.424 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:23.772 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:26.254 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:26.305 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:53.016 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:53.017 INFO:teuthology.orchestra.run.smithi117.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:24:53.017 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:24:53.060 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:53.448 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:53.448 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:24:53.448 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:53.448 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:53.878 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:53.878 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:24:53.878 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:24:53.878 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:24:53.878 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:24:53.878 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:24:53.878 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:24:53.878 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:24:53.878 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:24:54.216 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:24:54.325 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:24:54.326 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:24:54.326 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:24:54.326 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:24:54.326 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:24:54.326 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:24:54.327 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:24:54.327 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:24:54.406 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:24:54.406 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:24:54.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-03-29T10:24:54.449 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 68.el9 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8264254 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:24:54.450 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:24:54.451 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:24:54.452 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:24:54.452 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:24:54.452 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:24:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-03-29T10:24:56.399 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:24:56.456 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-03-29T10:24:56.458 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-29T10:24:56.473 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.5ACnbRMAp2 2024-03-29T10:24:56.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5ACnbRMAp2 2024-03-29T10:24:56.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.5ACnbRMAp2 2024-03-29T10:24:56.679 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.5ACnbRMAp2 is 6KB 2024-03-29T10:24:56.728 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.5ACnbRMAp2 2024-03-29T10:24:56.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:24:56.808 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:24:56.808 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:24:56.808 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:24:56.808 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:24:56.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:24:56.957 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-03-29T10:25:00.534 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:00.534 INFO:teuthology.orchestra.run.smithi136.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:25:00.534 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:25:00.580 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:00.955 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:00.955 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:25:00.955 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:25:00.955 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:25:01.366 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:01.366 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:25:01.367 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:25:01.367 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:01.367 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:25:01.367 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:25:01.367 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:25:01.367 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:25:01.367 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:25:01.707 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:25:01.802 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:01.802 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:25:01.802 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:25:01.803 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:25:01.803 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:25:01.803 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:25:01.803 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:25:01.803 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:25:01.883 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:01.884 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:25:01.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo rpm -qi grub2-tools 2024-03-29T10:25:01.925 INFO:teuthology.orchestra.run.smithi136.stdout:Name : grub2-tools 2024-03-29T10:25:01.925 INFO:teuthology.orchestra.run.smithi136.stdout:Epoch : 1 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Version : 2.06 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Release : 68.el9 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: x86_64 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Group : Unspecified 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Size : 8264254 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:License : GPLv3+ 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Packager : builder@centos.org 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:Vendor : CentOS 2024-03-29T10:25:01.926 INFO:teuthology.orchestra.run.smithi136.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:25:01.927 INFO:teuthology.orchestra.run.smithi136.stdout:Summary : Support tools for GRUB. 2024-03-29T10:25:01.927 INFO:teuthology.orchestra.run.smithi136.stdout:Description : 2024-03-29T10:25:01.927 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:25:01.927 INFO:teuthology.orchestra.run.smithi136.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:25:01.927 INFO:teuthology.orchestra.run.smithi136.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:25:01.927 INFO:teuthology.orchestra.run.smithi136.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:25:01.927 INFO:teuthology.orchestra.run.smithi136.stdout:hardware devices. 2024-03-29T10:25:01.927 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:25:01.927 INFO:teuthology.orchestra.run.smithi136.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:25:01.928 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:25:01.928 INFO:teuthology.task.kernel:Updating grub on smithi136 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:25:01.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:25:02.506 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-03-29T10:25:03.973 INFO:teuthology.orchestra.run.smithi136.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:25:04.022 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-03-29T10:25:04.024 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-03-29T10:25:04.040 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.g5QIsLtncz 2024-03-29T10:25:04.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.g5QIsLtncz 2024-03-29T10:25:04.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.g5QIsLtncz 2024-03-29T10:25:04.248 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.g5QIsLtncz is 6KB 2024-03-29T10:25:04.296 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.g5QIsLtncz 2024-03-29T10:25:04.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:25:04.374 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:25:04.375 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:25:04.375 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:25:04.375 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:25:04.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:25:04.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-03-29T10:25:26.981 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:25:26.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T10:25:26.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:25:34.534 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:25:34.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T10:25:34.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:25:45.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-03-29T10:25:52.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-29T10:25:54.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T10:25:54.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:25:57.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-03-29T10:26:01.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T10:26:01.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:04.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-29T10:26:09.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T10:26:09.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:16.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T10:26:16.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:20.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-29T10:26:27.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-03-29T10:26:35.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T10:26:35.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:38.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-29T10:26:43.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T10:26:43.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:43.342 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-29T10:26:43.741 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T10:26:43.741 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:26:43.741 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-29T10:26:43.755 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:26:43.755 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:26:43.756 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:26:43.756 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:26:44.756 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:26:44.756 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:26:44.818 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-03-29T10:26:44.845 DEBUG:teuthology.parallel:result is None 2024-03-29T10:26:56.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T10:26:56.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:56.536 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-29T10:26:56.903 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T10:26:56.903 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:26:56.903 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-29T10:26:56.919 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:26:56.919 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:26:56.919 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:26:56.920 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:26:57.920 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:26:57.921 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:26:58.013 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-03-29T10:26:58.057 DEBUG:teuthology.parallel:result is None 2024-03-29T10:26:58.057 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:26:58.067 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:26:58.067 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:26:58.070 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:26:58.088 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:26:58.095 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:26:58.106 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:26:58.107 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:26:58.129 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:26:58.235 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:26:58.244 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:26:58.244 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:26:58.246 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:26:58.276 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:26:58.285 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:26:58.290 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:26:58.299 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:26:58.300 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:26:58.308 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:26:58.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:26:58.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:26:58.370 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:26:58.379 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:26:58.380 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:26:58.404 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:26:58.426 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:26:58.502 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:26:58.553 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:26:58.635 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:26:58.678 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:26:58.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:26:58.745 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:26:58.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:26:58.843 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:26:58.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:26:58.974 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:26:58.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:26:59.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-03-29T10:26:59.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-03-29T10:26:59.083 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:26:59.141 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:26:59.604 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:26:59.612 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:26:59.612 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:26:59.629 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:26:59.642 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-03-29T10:26:59.713 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-03-29T10:26:59.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-03-29T10:26:59.784 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-03-29T10:26:59.786 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:26:59.786 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-03-29T10:26:59.816 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-03-29T10:26:59.816 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-03-29T10:26:59.850 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-03-29T10:26:59.850 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi117.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:26:59.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:26:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:26:59.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:26:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:26:59.920 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:26:59.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:26:59.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:27:00.002 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:27:00.012 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:27:00.030 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:27:00.032 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi117.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:31:13.461 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-03-29T10:31:13.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T10:31:13.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:31:13.554 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-29T10:31:13.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T10:31:13.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T10:31:13.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:31:13.718 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-29T10:31:13.797 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-29T10:31:13.797 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:31:13.809 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:31:13.810 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:31:13.810 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:31:13.812 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:31:13.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:31:13.842 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:31:13.859 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:31:13.868 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:31:13.885 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:31:13.888 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-03-29T10:31:13.902 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-03-29T10:31:13.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-03-29T10:31:13.918 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:31:13.928 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-03-29T10:31:13.934 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:31:13.944 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:31:13.962 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:31:13.985 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:31:13.989 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:31:13.989 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-29T10:31:13.989 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:13.989 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:13.989 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:13.989 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:14.021 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:31:14.025 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:31:14.025 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-29T10:31:14.025 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:14.025 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:14.025 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:14.025 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:14.025 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:31:14.034 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:31:14.034 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:31:14.034 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:31:14.034 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:31:14.061 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:31:14.061 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:31:14.062 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:31:14.062 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:31:14.064 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:31:14.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:31:14.211 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:31:14.212 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:31:14.282 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:31:14.282 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:31:14.383 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:31:14.383 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:31:14.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:31:14.411 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:31:14.411 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:31:14.454 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:31:14.454 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:31:14.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:31:14.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:31:14.484 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:31:14.484 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:31:14.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:31:14.563 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:31:14.629 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-03-29T10:31:14.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-03-29T10:31:14.632 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:31:14.698 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-03-29T10:31:14.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-03-29T10:31:14.882 INFO:teuthology.orchestra.run.smithi117.stdout:74 files removed 2024-03-29T10:31:14.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw 2024-03-29T10:31:14.936 INFO:teuthology.orchestra.run.smithi136.stdout:74 files removed 2024-03-29T10:31:14.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw 2024-03-29T10:31:15.612 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 234 kB/s | 82 kB 00:00 2024-03-29T10:31:15.692 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 169 kB/s | 82 kB 00:00 2024-03-29T10:31:16.042 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-03-29T10:31:16.054 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-29T10:31:16.485 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-29T10:31:16.505 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-29T10:31:16.904 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T10:31:16.966 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:31:18.377 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-29T10:31:18.579 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-29T10:31:19.165 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-29T10:31:19.678 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T10:31:21.218 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-29T10:31:22.112 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T10:31:25.195 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-29T10:31:26.287 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-29T10:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-29T10:31:27.858 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-29T10:31:29.100 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-29T10:31:30.276 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 21 MB 00:03 2024-03-29T10:31:34.039 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-29T10:31:35.242 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T10:31:36.659 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:31:36.663 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:31:36.663 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:31:36.664 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:31:36.665 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:31:36.666 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-03-29T10:31:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:31:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:31:36.667 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:31:36.667 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:31:36.667 INFO:teuthology.orchestra.run.smithi117.stdout:Install 30 Packages 2024-03-29T10:31:36.667 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 2 Packages 2024-03-29T10:31:36.667 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:31:36.667 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 58 M 2024-03-29T10:31:36.668 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:31:37.531 INFO:teuthology.orchestra.run.smithi117.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 8.1 MB/s | 5.3 MB 00:00 2024-03-29T10:31:37.606 INFO:teuthology.orchestra.run.smithi117.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 336 kB/s | 25 kB 00:00 2024-03-29T10:31:37.715 INFO:teuthology.orchestra.run.smithi117.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.7 MB/s | 742 kB 00:00 2024-03-29T10:31:37.822 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:31:37.826 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:31:37.826 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:31:37.826 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:31:37.826 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:31:37.826 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:31:37.826 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-03-29T10:31:37.826 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:31:37.826 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:31:37.827 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:31:37.828 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout:Install 30 Packages 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-03-29T10:31:37.829 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:31:37.830 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 58 M 2024-03-29T10:31:37.830 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:31:37.831 INFO:teuthology.orchestra.run.smithi117.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.2 MB/s | 498 kB 00:00 2024-03-29T10:31:39.224 INFO:teuthology.orchestra.run.smithi117.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 3.7 MB/s | 5.1 MB 00:01 2024-03-29T10:31:39.432 INFO:teuthology.orchestra.run.smithi117.stdout:(6/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 7.8 MB/s | 20 MB 00:02 2024-03-29T10:31:39.449 INFO:teuthology.orchestra.run.smithi117.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 200 kB/s | 45 kB 00:00 2024-03-29T10:31:39.499 INFO:teuthology.orchestra.run.smithi117.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T10:31:39.501 INFO:teuthology.orchestra.run.smithi136.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.3 MB/s | 5.3 MB 00:01 2024-03-29T10:31:39.524 INFO:teuthology.orchestra.run.smithi117.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-29T10:31:39.593 INFO:teuthology.orchestra.run.smithi136.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-29T10:31:39.616 INFO:teuthology.orchestra.run.smithi117.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 320 kB 00:00 2024-03-29T10:31:39.683 INFO:teuthology.orchestra.run.smithi117.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.8 MB/s | 298 kB 00:00 2024-03-29T10:31:39.709 INFO:teuthology.orchestra.run.smithi117.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-29T10:31:39.735 INFO:teuthology.orchestra.run.smithi136.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.1 MB/s | 742 kB 00:00 2024-03-29T10:31:39.810 INFO:teuthology.orchestra.run.smithi136.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 6.5 MB/s | 498 kB 00:00 2024-03-29T10:31:39.825 INFO:teuthology.orchestra.run.smithi117.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-03-29T10:31:39.851 INFO:teuthology.orchestra.run.smithi117.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-29T10:31:39.893 INFO:teuthology.orchestra.run.smithi117.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 660 kB/s | 44 kB 00:00 2024-03-29T10:31:39.951 INFO:teuthology.orchestra.run.smithi117.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-03-29T10:31:39.976 INFO:teuthology.orchestra.run.smithi117.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-29T10:31:40.002 INFO:teuthology.orchestra.run.smithi117.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-29T10:31:40.027 INFO:teuthology.orchestra.run.smithi117.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T10:31:40.052 INFO:teuthology.orchestra.run.smithi117.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-29T10:31:40.078 INFO:teuthology.orchestra.run.smithi117.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 918 kB/s | 46 kB 00:00 2024-03-29T10:31:40.178 INFO:teuthology.orchestra.run.smithi117.stdout:(22/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.9 MB/s | 9.7 MB 00:03 2024-03-29T10:31:40.320 INFO:teuthology.orchestra.run.smithi117.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-03-29T10:31:40.427 INFO:teuthology.orchestra.run.smithi136.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.3 MB/s | 5.1 MB 00:00 2024-03-29T10:31:40.470 INFO:teuthology.orchestra.run.smithi117.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 740 kB/s | 309 kB 00:00 2024-03-29T10:31:40.487 INFO:teuthology.orchestra.run.smithi117.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 291 kB/s | 49 kB 00:00 2024-03-29T10:31:40.494 INFO:teuthology.orchestra.run.smithi136.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 679 kB/s | 45 kB 00:00 2024-03-29T10:31:40.569 INFO:teuthology.orchestra.run.smithi136.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-29T10:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-29T10:31:40.705 INFO:teuthology.orchestra.run.smithi117.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 288 kB/s | 67 kB 00:00 2024-03-29T10:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 4.2 MB/s | 320 kB 00:00 2024-03-29T10:31:40.811 INFO:teuthology.orchestra.run.smithi136.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.2 MB/s | 298 kB 00:00 2024-03-29T10:31:40.886 INFO:teuthology.orchestra.run.smithi136.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-29T10:31:40.971 INFO:teuthology.orchestra.run.smithi117.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-03-29T10:31:40.987 INFO:teuthology.orchestra.run.smithi136.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-29T10:31:40.996 INFO:teuthology.orchestra.run.smithi117.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 657 kB/s | 192 kB 00:00 2024-03-29T10:31:41.022 INFO:teuthology.orchestra.run.smithi117.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 838 kB 00:00 2024-03-29T10:31:41.037 INFO:teuthology.orchestra.run.smithi136.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T10:31:41.087 INFO:teuthology.orchestra.run.smithi136.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-29T10:31:41.154 INFO:teuthology.orchestra.run.smithi136.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T10:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-29T10:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T10:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T10:31:41.305 INFO:teuthology.orchestra.run.smithi136.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T10:31:41.339 INFO:teuthology.orchestra.run.smithi136.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T10:31:41.523 INFO:teuthology.orchestra.run.smithi136.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-29T10:31:41.689 INFO:teuthology.orchestra.run.smithi136.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2024-03-29T10:31:41.740 INFO:teuthology.orchestra.run.smithi136.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 518 kB/s | 26 kB 00:00 2024-03-29T10:31:41.764 INFO:teuthology.orchestra.run.smithi117.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.2 MB/s | 3.1 MB 00:00 2024-03-29T10:31:41.790 INFO:teuthology.orchestra.run.smithi136.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 975 kB/s | 49 kB 00:00 2024-03-29T10:31:41.822 INFO:teuthology.orchestra.run.smithi117.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-03-29T10:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-29T10:31:41.872 INFO:teuthology.orchestra.run.smithi117.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.9 MB/s | 3.4 MB 00:00 2024-03-29T10:31:41.876 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:31:41.876 INFO:teuthology.orchestra.run.smithi117.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-29T10:31:41.877 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2024-03-29T10:31:41.912 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:31:41.912 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:31:41.924 INFO:teuthology.orchestra.run.smithi136.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-03-29T10:31:41.983 INFO:teuthology.orchestra.run.smithi136.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-03-29T10:31:42.375 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:31:42.376 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:31:42.708 INFO:teuthology.orchestra.run.smithi136.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.7 MB/s | 3.4 MB 00:00 2024-03-29T10:31:43.367 INFO:teuthology.orchestra.run.smithi136.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.8 MB/s | 3.1 MB 00:00 2024-03-29T10:31:43.821 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:31:43.903 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:31:43.935 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:31:43.960 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:31:43.983 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:31:44.000 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:31:44.192 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:31:44.246 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:31:44.308 INFO:teuthology.orchestra.run.smithi136.stdout:(31/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 1.5 MB/s | 9.7 MB 00:06 2024-03-29T10:31:44.579 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:31:44.645 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:31:44.720 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:31:44.822 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:31:44.880 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:31:44.926 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:31:44.955 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:31:44.999 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:31:45.067 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:31:45.090 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:31:45.154 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:31:45.181 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:31:45.280 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:31:45.323 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:31:45.355 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:31:45.388 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:31:45.417 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:31:45.450 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:31:45.775 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:31:45.805 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:31:45.877 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:31:45.910 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:31:45.953 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:31:46.406 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:31:46.432 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:31:46.505 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:31:46.570 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:31:46.593 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:31:48.709 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:31:48.756 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:31:49.338 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:31:49.375 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:31:49.463 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:31:49.463 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:31:49.464 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:31:49.464 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:31:49.502 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:31:50.816 INFO:teuthology.orchestra.run.smithi136.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 20 MB 00:12 2024-03-29T10:31:50.819 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:31:50.819 INFO:teuthology.orchestra.run.smithi136.stdout:Total 4.5 MB/s | 58 MB 00:12 2024-03-29T10:31:50.820 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:31:50.857 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:31:50.857 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:31:51.334 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:31:51.334 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:31:52.784 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:31:52.861 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:31:52.887 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:31:52.912 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:31:52.934 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:31:52.952 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:31:53.234 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:31:53.288 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:31:53.675 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:31:53.705 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:31:53.797 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:31:53.886 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:31:53.915 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:31:53.961 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:31:53.990 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:31:54.034 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:31:54.102 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:31:54.317 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:31:54.367 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:31:54.407 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:31:54.440 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:31:54.468 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:31:54.501 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:31:54.856 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:31:54.946 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:31:54.977 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:31:55.018 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:31:55.469 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:31:55.500 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:31:55.588 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:31:55.656 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:31:55.686 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:31:57.849 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:31:57.882 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:31:58.431 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:31:58.468 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:31:58.559 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:31:58.559 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:31:58.560 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:31:58.560 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:31:58.602 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:31:59.213 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:31:59.213 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-03-29T10:31:59.213 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-03-29T10:31:59.213 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-03-29T10:31:59.213 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-03-29T10:31:59.214 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-03-29T10:31:59.214 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-03-29T10:31:59.214 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-03-29T10:31:59.214 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:00.154 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:00.178 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:00.178 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:00.178 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:32:00.179 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:00.179 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:00.179 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:00.180 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:00.264 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:00.264 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:32:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:32:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:32:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:32:01.337 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:32:01.338 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:32:01.339 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:32:01.340 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:01.340 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:32:01.340 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:32:02.297 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:02.297 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:02.297 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-03-29T10:32:02.297 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.297 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.297 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.298 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:32:02.299 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:32:02.300 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:32:02.300 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:02.300 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:32:02.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-test 2024-03-29T10:32:03.190 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:32:03.349 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:32:03.350 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:32:03.350 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:32:03.350 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:32:03.350 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:32:03.350 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:32:03.350 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:32:03.350 INFO:teuthology.orchestra.run.smithi117.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:32:03.350 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:32:03.351 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:32:03.351 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:32:03.351 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:32:03.351 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:03.351 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:32:03.351 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:32:03.351 INFO:teuthology.orchestra.run.smithi117.stdout:Install 6 Packages 2024-03-29T10:32:03.351 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:03.352 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 48 M 2024-03-29T10:32:03.352 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 204 M 2024-03-29T10:32:03.352 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:32:03.552 INFO:teuthology.orchestra.run.smithi117.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:32:03.602 INFO:teuthology.orchestra.run.smithi117.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:32:03.636 INFO:teuthology.orchestra.run.smithi117.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:32:03.761 INFO:teuthology.orchestra.run.smithi117.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 439 kB/s | 163 kB 00:00 2024-03-29T10:32:03.786 INFO:teuthology.orchestra.run.smithi117.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T10:32:08.087 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:08.087 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-03-29T10:32:08.087 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-03-29T10:32:08.087 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-03-29T10:32:08.087 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-03-29T10:32:08.087 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-03-29T10:32:08.087 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-03-29T10:32:08.087 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-03-29T10:32:08.087 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:08.803 INFO:teuthology.orchestra.run.smithi117.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 8.7 MB/s | 47 MB 00:05 2024-03-29T10:32:08.805 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:08.805 INFO:teuthology.orchestra.run.smithi117.stdout:Total 8.8 MB/s | 48 MB 00:05 2024-03-29T10:32:08.819 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:32:08.837 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:32:08.837 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:32:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:09.024 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:09.024 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:09.024 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:32:09.024 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:09.024 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:09.024 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:09.026 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:09.110 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:09.110 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:09.220 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:32:09.221 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:32:09.793 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:32:09.878 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:32:09.915 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:32:09.946 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:09.990 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:32:10.014 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:32:10.292 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:32:10.293 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:32:10.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:10.295 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:32:10.295 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:32:11.563 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:32:11.564 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:32:11.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:32:11.566 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:11.567 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:32:11.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-test 2024-03-29T10:32:12.375 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:37 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:32:12.524 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:32:12.525 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:32:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:32:12.525 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:32:12.525 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:32:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-03-29T10:32:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:12.527 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 48 M 2024-03-29T10:32:12.527 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 204 M 2024-03-29T10:32:12.527 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:32:12.800 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:32:12.850 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:32:12.884 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:32:12.976 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 488 kB/s | 163 kB 00:00 2024-03-29T10:32:13.075 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 982 kB/s | 187 kB 00:00 2024-03-29T10:32:14.514 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:14.605 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:15.343 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-29T10:32:15.346 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:15.346 INFO:teuthology.orchestra.run.smithi136.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-29T10:32:15.362 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:32:15.383 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:32:15.383 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:32:15.766 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:32:15.766 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:32:16.346 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:32:16.411 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:32:16.457 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:32:16.488 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:16.529 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:32:16.554 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:16.985 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:16.985 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:32:16.985 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:32:16.985 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:16.985 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:32:16.985 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:17.357 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:32:17.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph 2024-03-29T10:32:18.281 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:44 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:32:18.434 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:32:18.437 INFO:teuthology.orchestra.run.smithi117.stdout:========================================================================================= 2024-03-29T10:32:18.437 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:32:18.437 INFO:teuthology.orchestra.run.smithi117.stdout:========================================================================================= 2024-03-29T10:32:18.437 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:32:18.437 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:32:18.437 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:32:18.437 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:32:18.437 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:32:18.438 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:32:18.439 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout:========================================================================================= 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout:Install 36 Packages 2024-03-29T10:32:18.440 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:18.441 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 30 M 2024-03-29T10:32:18.441 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 107 M 2024-03-29T10:32:18.441 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:32:18.964 INFO:teuthology.orchestra.run.smithi117.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-29T10:32:19.247 INFO:teuthology.orchestra.run.smithi117.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2024-03-29T10:32:19.872 INFO:teuthology.orchestra.run.smithi117.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 2.1 MB 00:01 2024-03-29T10:32:20.073 INFO:teuthology.orchestra.run.smithi117.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.2 MB/s | 247 kB 00:00 2024-03-29T10:32:20.206 INFO:teuthology.orchestra.run.smithi117.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T10:32:20.248 INFO:teuthology.orchestra.run.smithi117.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:32:20.273 INFO:teuthology.orchestra.run.smithi117.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T10:32:20.299 INFO:teuthology.orchestra.run.smithi117.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-29T10:32:20.332 INFO:teuthology.orchestra.run.smithi117.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:32:20.408 INFO:teuthology.orchestra.run.smithi117.stdout:(10/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.2 MB/s | 4.7 MB 00:01 2024-03-29T10:32:20.441 INFO:teuthology.orchestra.run.smithi117.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-29T10:32:20.510 INFO:teuthology.orchestra.run.smithi117.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-29T10:32:20.533 INFO:teuthology.orchestra.run.smithi117.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-29T10:32:20.559 INFO:teuthology.orchestra.run.smithi117.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:32:20.584 INFO:teuthology.orchestra.run.smithi117.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T10:32:20.759 INFO:teuthology.orchestra.run.smithi117.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 11 MB/s | 17 MB 00:01 2024-03-29T10:32:20.826 INFO:teuthology.orchestra.run.smithi117.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2024-03-29T10:32:20.951 INFO:teuthology.orchestra.run.smithi117.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 318 kB/s | 125 kB 00:00 2024-03-29T10:32:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:21.052 INFO:teuthology.orchestra.run.smithi117.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 193 kB/s | 43 kB 00:00 2024-03-29T10:32:21.085 INFO:teuthology.orchestra.run.smithi117.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 464 kB/s | 151 kB 00:00 2024-03-29T10:32:21.113 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:21.145 INFO:teuthology.orchestra.run.smithi117.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T10:32:21.203 INFO:teuthology.orchestra.run.smithi117.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-29T10:32:21.236 INFO:teuthology.orchestra.run.smithi117.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 603 kB/s | 172 kB 00:00 2024-03-29T10:32:21.262 INFO:teuthology.orchestra.run.smithi117.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 393 kB/s | 23 kB 00:00 2024-03-29T10:32:21.295 INFO:teuthology.orchestra.run.smithi117.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-03-29T10:32:21.320 INFO:teuthology.orchestra.run.smithi117.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-03-29T10:32:21.354 INFO:teuthology.orchestra.run.smithi117.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 791 kB/s | 46 kB 00:00 2024-03-29T10:32:21.388 INFO:teuthology.orchestra.run.smithi117.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-29T10:32:21.413 INFO:teuthology.orchestra.run.smithi117.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 855 kB/s | 79 kB 00:00 2024-03-29T10:32:21.447 INFO:teuthology.orchestra.run.smithi117.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-29T10:32:21.489 INFO:teuthology.orchestra.run.smithi117.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-29T10:32:21.505 INFO:teuthology.orchestra.run.smithi117.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-03-29T10:32:21.531 INFO:teuthology.orchestra.run.smithi117.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 766 kB/s | 90 kB 00:00 2024-03-29T10:32:21.556 INFO:teuthology.orchestra.run.smithi117.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-03-29T10:32:21.589 INFO:teuthology.orchestra.run.smithi117.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-29T10:32:21.781 INFO:teuthology.orchestra.run.smithi117.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.5 MB/s | 427 kB 00:00 2024-03-29T10:32:21.784 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:21.784 INFO:teuthology.orchestra.run.smithi117.stdout:Total 8.9 MB/s | 30 MB 00:03 2024-03-29T10:32:21.809 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:32:21.850 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:32:21.851 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:32:22.265 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:32:22.265 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:32:23.599 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:23.599 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:32:23.599 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:32:23.599 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:23.599 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:32:23.599 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:32:23.623 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:32:23.692 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:32:23.732 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:32:23.761 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:32:23.832 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:32:23.885 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:32:23.917 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:32:24.010 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:32:24.088 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:32:24.129 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:32:24.201 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:32:24.254 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:32:24.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph 2024-03-29T10:32:24.292 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:32:24.342 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:32:24.398 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:32:24.587 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:32:24.630 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:32:24.660 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:32:24.695 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:32:24.725 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:32:24.801 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:32:24.844 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:32:24.856 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:49 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:32:24.877 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:32:24.917 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:32:24.952 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:32:24.995 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:32:25.019 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:32:25.022 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-29T10:32:25.022 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:32:25.022 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-29T10:32:25.022 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:32:25.022 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:32:25.022 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:32:25.022 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:32:25.023 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:32:25.024 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout:Install 36 Packages 2024-03-29T10:32:25.025 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:25.026 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 30 M 2024-03-29T10:32:25.026 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 107 M 2024-03-29T10:32:25.026 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:32:25.085 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:32:25.134 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:32:25.207 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:32:25.345 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:32:25.469 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:32:25.636 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:32:25.664 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:32:25.689 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:32:25.690 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:25.690 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:32:25.690 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:32:25.690 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:32:25.690 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:25.972 INFO:teuthology.orchestra.run.smithi136.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 26 kB/s | 6.6 kB 00:00 2024-03-29T10:32:26.264 INFO:teuthology.orchestra.run.smithi136.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.9 MB/s | 2.1 MB 00:00 2024-03-29T10:32:26.298 INFO:teuthology.orchestra.run.smithi136.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 1.5 MB 00:00 2024-03-29T10:32:26.398 INFO:teuthology.orchestra.run.smithi136.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-29T10:32:26.515 INFO:teuthology.orchestra.run.smithi136.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T10:32:26.607 INFO:teuthology.orchestra.run.smithi136.stdout:(6/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 7.4 MB/s | 4.7 MB 00:00 2024-03-29T10:32:26.632 INFO:teuthology.orchestra.run.smithi136.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-29T10:32:26.649 INFO:teuthology.orchestra.run.smithi136.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-03-29T10:32:26.666 INFO:teuthology.orchestra.run.smithi136.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:32:26.683 INFO:teuthology.orchestra.run.smithi136.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:32:26.717 INFO:teuthology.orchestra.run.smithi136.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:32:26.776 INFO:teuthology.orchestra.run.smithi136.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-03-29T10:32:26.792 INFO:teuthology.orchestra.run.smithi136.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T10:32:26.818 INFO:teuthology.orchestra.run.smithi136.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T10:32:26.843 INFO:teuthology.orchestra.run.smithi136.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T10:32:26.927 INFO:teuthology.orchestra.run.smithi136.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-29T10:32:26.985 INFO:teuthology.orchestra.run.smithi136.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-03-29T10:32:27.124 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:32:27.151 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:32:27.151 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:27.151 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:32:27.151 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:32:27.151 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:32:27.151 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:27.369 INFO:teuthology.orchestra.run.smithi136.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 15 MB/s | 17 MB 00:01 2024-03-29T10:32:27.494 INFO:teuthology.orchestra.run.smithi136.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 266 kB/s | 151 kB 00:00 2024-03-29T10:32:27.511 INFO:teuthology.orchestra.run.smithi136.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 87 kB/s | 43 kB 00:00 2024-03-29T10:32:27.570 INFO:teuthology.orchestra.run.smithi136.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 863 kB/s | 172 kB 00:00 2024-03-29T10:32:27.591 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:32:27.614 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:32:27.615 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:27.615 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:32:27.615 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:32:27.615 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:32:27.615 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-29T10:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-03-29T10:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-03-29T10:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-03-29T10:32:27.713 INFO:teuthology.orchestra.run.smithi136.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-03-29T10:32:27.730 INFO:teuthology.orchestra.run.smithi136.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-29T10:32:27.747 INFO:teuthology.orchestra.run.smithi136.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 689 kB/s | 46 kB 00:00 2024-03-29T10:32:27.780 INFO:teuthology.orchestra.run.smithi136.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-29T10:32:27.823 INFO:teuthology.orchestra.run.smithi136.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-29T10:32:27.825 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:32:27.848 INFO:teuthology.orchestra.run.smithi136.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 162 kB/s | 16 kB 00:00 2024-03-29T10:32:27.848 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:32:27.848 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:27.848 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:32:27.848 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:32:27.849 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:32:27.849 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:27.873 INFO:teuthology.orchestra.run.smithi136.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 977 kB/s | 90 kB 00:00 2024-03-29T10:32:27.898 INFO:teuthology.orchestra.run.smithi136.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-03-29T10:32:27.914 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:32:27.924 INFO:teuthology.orchestra.run.smithi136.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T10:32:27.965 INFO:teuthology.orchestra.run.smithi136.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-29T10:32:27.991 INFO:teuthology.orchestra.run.smithi136.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-29T10:32:27.997 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:27.997 INFO:teuthology.orchestra.run.smithi136.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-29T10:32:28.022 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:32:28.064 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:32:28.065 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:32:28.476 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:32:28.477 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:32:29.635 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:32:29.635 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:32:29.635 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:32:29.635 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:32:29.635 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:32:29.635 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:32:29.635 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:32:29.635 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:32:29.635 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:32:29.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:32:29.637 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:32:29.637 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:32:29.637 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:32:29.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:32:29.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:32:29.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:32:29.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:32:29.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:32:29.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:32:29.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:32:29.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:32:29.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:32:29.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:32:29.837 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:32:29.902 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:32:29.937 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:32:29.971 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:32:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:32:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:32:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:32:30.211 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:32:30.281 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:32:30.322 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:32:30.395 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:32:30.418 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:32:30.418 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:30.418 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:32:30.418 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:32:30.419 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:32:30.420 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:32:30.421 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:32:30.421 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:32:30.421 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:32:30.421 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:32:30.421 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:32:30.421 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:30.421 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:32:30.448 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:32:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:32:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:32:30.590 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:32:30.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base 2024-03-29T10:32:30.777 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:32:30.832 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:32:30.871 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:32:30.914 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:32:30.952 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:32:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:32:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:32:31.088 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:32:31.128 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:32:31.163 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:32:31.198 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:32:31.288 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:32:31.326 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:32:31.329 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:32:31.401 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:32:31.424 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:32:31.476 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:32:31.477 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T10:32:31.477 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:32:31.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install cephadm 2024-03-29T10:32:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:32:31.673 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:32:31.830 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:32:31.859 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:32:31.885 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:32:31.886 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:31.886 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:32:31.886 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:32:31.886 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:32:31.886 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:32.103 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:32:32.254 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:32:32.255 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:32:32.255 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:32:32.255 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:32:32.255 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:32:32.255 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:32:32.255 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:32.255 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:32:32.255 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:32:32.255 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T10:32:32.256 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:32.256 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 747 k 2024-03-29T10:32:32.256 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 754 k 2024-03-29T10:32:32.256 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:32:32.548 INFO:teuthology.orchestra.run.smithi117.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.5 MB/s | 747 kB 00:00 2024-03-29T10:32:32.549 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:32.549 INFO:teuthology.orchestra.run.smithi117.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-29T10:32:32.549 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:32:32.554 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:32:32.554 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:32:32.560 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:32:32.560 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:32:32.636 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:32:33.089 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:32:33.187 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:32:33.318 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:32:33.345 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:32:33.345 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:33.345 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:32:33.345 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:32:33.345 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:32:33.345 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:33.551 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:32:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:32:33.809 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:32:33.809 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:33.809 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:32:33.809 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:32:33.809 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:32:33.809 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:33.847 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:32:33.847 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:33.847 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:32:33.847 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:33.847 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:33.847 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:32:34.011 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:32:34.035 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:32:34.035 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:34.035 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:32:34.035 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:32:34.035 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:32:34.035 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:34.073 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:32:34.092 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:32:34.638 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:32:34.791 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:32:34.792 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:32:34.792 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-29T10:32:34.792 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:32:34.792 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:32:34.792 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:32:34.792 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:34.792 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:32:34.792 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:32:34.792 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T10:32:34.793 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:34.793 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 145 k 2024-03-29T10:32:34.793 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 436 k 2024-03-29T10:32:34.793 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:32:35.314 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 279 kB/s | 145 kB 00:00 2024-03-29T10:32:35.315 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:35.315 INFO:teuthology.orchestra.run.smithi117.stdout:Total 278 kB/s | 145 kB 00:00 2024-03-29T10:32:35.315 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:32:35.322 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:32:35.323 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:32:35.376 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:32:35.376 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:32:35.519 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:32:35.590 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:32:35.614 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:32:35.615 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:35.615 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:32:35.615 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:35.814 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:32:35.815 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:32:35.816 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:32:35.817 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:32:35.817 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:32:35.817 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:32:35.818 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:32:35.818 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:32:35.818 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:32:35.818 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:32:35.818 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:32:35.818 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:32:35.818 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:32:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:32:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:36.587 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:32:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:36.587 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:36.588 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:32:36.604 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:32:36.605 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:32:36.606 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:32:36.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr 2024-03-29T10:32:36.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base 2024-03-29T10:32:37.425 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:32:37.525 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:32:37.560 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:32:37.577 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:32:37.578 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T10:32:37.578 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:32:37.659 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:32:37.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:32:37.711 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:32:37.712 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T10:32:37.712 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:32:37.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install cephadm 2024-03-29T10:32:38.214 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:32:38.347 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:32:38.364 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:32:38.365 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-03-29T10:32:38.365 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:32:38.366 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:32:38.367 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:32:38.367 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:38.367 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:32:38.367 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-03-29T10:32:38.367 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-03-29T10:32:38.367 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:38.367 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 6.9 M 2024-03-29T10:32:38.367 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-03-29T10:32:38.368 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:32:38.498 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:38.499 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 747 k 2024-03-29T10:32:38.500 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 754 k 2024-03-29T10:32:38.500 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:32:38.821 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.3 MB/s | 747 kB 00:00 2024-03-29T10:32:38.822 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:38.822 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.3 MB/s | 747 kB 00:00 2024-03-29T10:32:38.822 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:32:38.827 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:32:38.827 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:32:38.833 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:32:38.833 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:32:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:32:39.176 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 175 kB/s | 31 kB 00:00 2024-03-29T10:32:39.326 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-29T10:32:39.343 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 44 kB/s | 15 kB 00:00 2024-03-29T10:32:39.366 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:32:39.368 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-29T10:32:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:32:39.510 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.5 MB/s | 1.8 MB 00:00 2024-03-29T10:32:39.602 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-03-29T10:32:39.831 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:32:39.911 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 49 kB/s | 19 kB 00:00 2024-03-29T10:32:40.111 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 738 kB/s | 548 kB 00:00 2024-03-29T10:32:40.131 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:32:40.132 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:40.132 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:32:40.132 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:40.132 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:40.132 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:32:40.219 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 283 kB/s | 31 kB 00:00 2024-03-29T10:32:40.245 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 433 kB/s | 144 kB 00:00 2024-03-29T10:32:40.345 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-29T10:32:40.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:32:40.537 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-03-29T10:32:40.541 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:40.542 INFO:teuthology.orchestra.run.smithi117.stdout:Total 3.2 MB/s | 6.9 MB 00:02 2024-03-29T10:32:40.559 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:32:40.576 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:32:40.576 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:32:40.749 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:32:40.750 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:32:40.950 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:32:41.106 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:32:41.106 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 145 k 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 436 k 2024-03-29T10:32:41.107 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:32:41.274 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:32:41.353 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:32:41.431 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:32:41.477 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:32:41.516 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:32:41.532 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 342 kB/s | 145 kB 00:00 2024-03-29T10:32:41.533 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:41.533 INFO:teuthology.orchestra.run.smithi136.stdout:Total 341 kB/s | 145 kB 00:00 2024-03-29T10:32:41.533 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:32:41.541 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:32:41.541 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:32:41.593 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:32:41.616 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:32:41.737 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:32:41.743 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:32:41.792 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:32:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:32:41.816 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:41.816 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:32:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:41.834 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:32:41.895 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:32:41.925 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:32:41.956 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:32:42.195 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:32:42.231 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:32:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:32:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:42.782 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:32:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:42.782 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:32:43.046 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:32:43.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:32:43.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:32:43.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:32:43.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:32:43.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:32:43.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:32:43.047 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:32:43.047 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:32:43.047 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:32:43.047 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:32:43.047 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:32:43.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr 2024-03-29T10:32:43.396 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:32:43.396 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:43.396 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:32:43.396 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:32:43.396 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:43.397 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:32:43.626 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:32:43.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:32:43.726 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:32:43.777 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:32:43.779 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T10:32:43.779 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:32:43.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:32:44.228 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:32:44.382 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:32:44.383 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-29T10:32:44.383 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:32:44.383 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-29T10:32:44.383 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:32:44.384 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:32:44.385 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:32:44.385 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:32:44.385 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:44.385 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:32:44.385 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-29T10:32:44.385 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-03-29T10:32:44.385 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:44.386 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 37 M 2024-03-29T10:32:44.386 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 209 M 2024-03-29T10:32:44.386 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:32:44.408 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:32:44.557 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:32:44.558 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-29T10:32:44.558 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:32:44.558 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-29T10:32:44.558 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:32:44.558 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:32:44.558 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-29T10:32:44.559 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-03-29T10:32:44.560 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:44.560 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 6.9 M 2024-03-29T10:32:44.560 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-03-29T10:32:44.560 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:32:44.759 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 32 kB 00:00 2024-03-29T10:32:44.793 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 518 kB/s | 17 kB 00:00 2024-03-29T10:32:44.827 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:32:44.994 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T10:32:45.135 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-29T10:32:45.177 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-29T10:32:45.261 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-03-29T10:32:45.504 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 36 kB/s | 15 kB 00:00 2024-03-29T10:32:45.553 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-29T10:32:45.612 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-29T10:32:45.688 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-29T10:32:45.713 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.3 MB/s | 238 kB 00:00 2024-03-29T10:32:45.737 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-29T10:32:45.762 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-29T10:32:45.922 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.6 MB/s | 548 kB 00:00 2024-03-29T10:32:45.955 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-03-29T10:32:45.972 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 387 kB/s | 19 kB 00:00 2024-03-29T10:32:46.040 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-29T10:32:46.090 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 613 kB/s | 31 kB 00:00 2024-03-29T10:32:46.132 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.7 MB/s | 1.8 MB 00:01 2024-03-29T10:32:46.157 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T10:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.6 MB/s | 2.0 MB 00:00 2024-03-29T10:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout:Total 4.0 MB/s | 6.9 MB 00:01 2024-03-29T10:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:32:46.362 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 393 kB/s | 245 kB 00:00 2024-03-29T10:32:46.469 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:32:46.470 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:32:46.977 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:32:47.049 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:32:47.093 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:32:47.140 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:32:47.170 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:32:47.266 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:32:47.386 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:32:47.473 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:32:47.525 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:32:47.554 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:32:47.585 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:32:47.795 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.3 MB/s | 7.4 MB 00:03 2024-03-29T10:32:47.798 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:47.798 INFO:teuthology.orchestra.run.smithi117.stdout:Total 11 MB/s | 37 MB 00:03 2024-03-29T10:32:47.827 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:32:47.842 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:32:47.861 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:32:47.868 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:32:47.868 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:32:48.225 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:32:48.225 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:32:48.648 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:32:48.648 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:32:48.648 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:32:48.649 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:32:48.649 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:32:48.649 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:32:48.649 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:32:48.649 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:32:48.649 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:32:48.649 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:32:48.649 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:32:48.649 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:32:48.931 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:32:49.001 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:32:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:32:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:49.004 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:32:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:32:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:32:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:32:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:32:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:32:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:32:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:32:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:32:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:32:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:49.006 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:32:49.041 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:32:49.150 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:32:49.202 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:32:49.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:32:49.379 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:32:49.410 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:32:49.494 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:32:49.810 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:14 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:32:49.964 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:32:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-03-29T10:32:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:49.967 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 37 M 2024-03-29T10:32:49.967 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 209 M 2024-03-29T10:32:49.967 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:32:50.134 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:32:50.240 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 226 kB/s | 32 kB 00:00 2024-03-29T10:32:50.246 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:32:50.265 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 717 kB/s | 17 kB 00:00 2024-03-29T10:32:50.299 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:32:50.366 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T10:32:50.625 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T10:32:50.650 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-03-29T10:32:50.951 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-29T10:32:51.042 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 7.8 MB/s | 7.4 MB 00:00 2024-03-29T10:32:51.068 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 803 kB 00:00 2024-03-29T10:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T10:32:51.471 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:32:51.985 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 261 kB/s | 245 kB 00:00 2024-03-29T10:32:51.988 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:51.988 INFO:teuthology.orchestra.run.smithi136.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-29T10:32:52.032 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:32:52.058 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:32:52.058 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:32:52.418 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:32:52.418 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:32:52.491 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:32:52.517 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:32:53.129 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:32:53.207 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:32:53.239 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:32:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:32:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:32:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:32:53.582 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:32:53.666 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:32:54.339 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:32:54.449 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:32:55.028 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:32:55.028 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:32:55.028 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:32:55.028 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:32:55.028 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:32:55.028 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:32:55.028 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:32:55.028 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:32:55.029 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:32:55.029 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:32:55.029 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:32:55.029 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:32:55.454 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:32:55.455 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:32:55.456 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:55.456 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:32:55.695 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:32:55.806 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:32:56.379 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:32:56.531 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:32:56.532 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T10:32:56.532 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:32:56.532 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T10:32:56.532 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:32:56.532 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:32:56.532 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:32:56.532 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:56.533 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:32:56.534 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T10:32:56.534 INFO:teuthology.orchestra.run.smithi117.stdout:Install 13 Packages 2024-03-29T10:32:56.534 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:32:56.534 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 2.0 M 2024-03-29T10:32:56.534 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-03-29T10:32:56.535 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:32:56.729 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:32:56.750 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:32:56.801 INFO:teuthology.orchestra.run.smithi117.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 199 kB/s | 24 kB 00:00 2024-03-29T10:32:56.826 INFO:teuthology.orchestra.run.smithi117.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-29T10:32:56.885 INFO:teuthology.orchestra.run.smithi117.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-29T10:32:56.910 INFO:teuthology.orchestra.run.smithi117.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-29T10:32:56.935 INFO:teuthology.orchestra.run.smithi117.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:32:56.961 INFO:teuthology.orchestra.run.smithi117.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 860 kB/s | 43 kB 00:00 2024-03-29T10:32:56.986 INFO:teuthology.orchestra.run.smithi117.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 170 kB/s | 52 kB 00:00 2024-03-29T10:32:57.311 INFO:teuthology.orchestra.run.smithi117.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 43 kB/s | 15 kB 00:00 2024-03-29T10:32:57.361 INFO:teuthology.orchestra.run.smithi117.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 76 kB/s | 32 kB 00:00 2024-03-29T10:32:57.537 INFO:teuthology.orchestra.run.smithi117.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-29T10:32:57.562 INFO:teuthology.orchestra.run.smithi117.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 352 kB/s | 203 kB 00:00 2024-03-29T10:32:57.679 INFO:teuthology.orchestra.run.smithi117.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 633 kB/s | 90 kB 00:00 2024-03-29T10:32:57.829 INFO:teuthology.orchestra.run.smithi117.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-29T10:32:57.833 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:57.833 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T10:32:57.834 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:32:57.847 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:32:57.847 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:32:58.452 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:32:58.619 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:32:58.690 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:32:58.753 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:32:58.790 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:32:58.840 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:32:58.934 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:32:59.034 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:32:59.092 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:32:59.289 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:32:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:32:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:32:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:32:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:32:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:32:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:32:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:32:59.698 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:32:59.808 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:32:59.855 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:32:59.888 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:32:59.937 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:32:59.958 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:00.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:33:00.629 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:00.692 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:00.692 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:33:00.692 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:33:00.692 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:33:00.692 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:33:00.692 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:33:00.692 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:33:00.692 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:33:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:33:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:33:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:33:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:33:00.693 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:33:00.781 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:00.782 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T10:33:00.782 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:33:00.782 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T10:33:00.782 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:33:00.782 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:33:00.783 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:33:00.784 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:33:00.784 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:00.784 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:33:00.784 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T10:33:00.784 INFO:teuthology.orchestra.run.smithi136.stdout:Install 13 Packages 2024-03-29T10:33:00.784 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:00.785 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 2.0 M 2024-03-29T10:33:00.785 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-03-29T10:33:00.785 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:33:00.961 INFO:teuthology.orchestra.run.smithi136.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 195 kB/s | 24 kB 00:00 2024-03-29T10:33:00.986 INFO:teuthology.orchestra.run.smithi136.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2024-03-29T10:33:01.036 INFO:teuthology.orchestra.run.smithi136.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:33:01.050 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:33:01.050 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:01.050 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:33:01.050 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:01.050 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:33:01.050 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:33:01.050 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:33:01.050 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:01.051 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:01.062 INFO:teuthology.orchestra.run.smithi136.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:33:01.087 INFO:teuthology.orchestra.run.smithi136.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:33:01.113 INFO:teuthology.orchestra.run.smithi136.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-29T10:33:01.138 INFO:teuthology.orchestra.run.smithi136.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 173 kB/s | 52 kB 00:00 2024-03-29T10:33:01.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:33:01.388 INFO:teuthology.orchestra.run.smithi136.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-29T10:33:01.413 INFO:teuthology.orchestra.run.smithi136.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-03-29T10:33:01.572 INFO:teuthology.orchestra.run.smithi136.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 467 kB/s | 203 kB 00:00 2024-03-29T10:33:01.597 INFO:teuthology.orchestra.run.smithi136.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-03-29T10:33:01.706 INFO:teuthology.orchestra.run.smithi136.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 671 kB/s | 90 kB 00:00 2024-03-29T10:33:01.739 INFO:teuthology.orchestra.run.smithi136.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.9 MB/s | 1.0 MB 00:00 2024-03-29T10:33:01.743 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:01.743 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-29T10:33:01.744 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:33:01.757 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:33:01.757 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:33:01.848 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:33:01.850 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:33:01.899 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:02.051 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:02.052 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T10:33:02.052 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:33:02.052 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T10:33:02.052 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:33:02.052 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:33:02.052 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:33:02.052 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:33:02.052 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:33:02.053 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:33:02.053 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:33:02.053 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:33:02.053 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:02.053 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:33:02.053 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T10:33:02.053 INFO:teuthology.orchestra.run.smithi117.stdout:Install 6 Packages 2024-03-29T10:33:02.053 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:02.054 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 6.8 M 2024-03-29T10:33:02.054 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 33 M 2024-03-29T10:33:02.054 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:33:02.396 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:33:02.562 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:33:02.616 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:33:02.664 INFO:teuthology.orchestra.run.smithi117.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-29T10:33:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:33:02.708 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:33:02.731 INFO:teuthology.orchestra.run.smithi117.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 432 kB/s | 138 kB 00:00 2024-03-29T10:33:02.750 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:33:02.756 INFO:teuthology.orchestra.run.smithi117.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 720 kB/s | 248 kB 00:00 2024-03-29T10:33:02.845 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:33:02.890 INFO:teuthology.orchestra.run.smithi117.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-03-29T10:33:02.914 INFO:teuthology.orchestra.run.smithi117.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-03-29T10:33:02.945 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:33:02.995 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:33:03.015 INFO:teuthology.orchestra.run.smithi117.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-29T10:33:03.017 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:03.017 INFO:teuthology.orchestra.run.smithi117.stdout:Total 7.0 MB/s | 6.8 MB 00:00 2024-03-29T10:33:03.031 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:33:03.042 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:33:03.042 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:33:03.129 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:33:03.130 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:33:03.429 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:33:03.607 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:33:03.649 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:33:03.726 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:33:03.784 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:33:03.818 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:33:03.867 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:33:03.896 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:04.170 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:33:04.261 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:33:04.322 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:33:04.345 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:04.624 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:04.624 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:33:04.624 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:33:04.624 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:33:04.624 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:33:04.624 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:33:04.624 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:33:04.625 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:33:04.625 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:33:04.625 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:33:04.625 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:33:04.625 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:33:04.625 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:33:04.995 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:33:04.995 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:33:04.996 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:33:04.997 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:04.997 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:05.111 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:05.111 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:33:05.112 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:33:05.112 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:33:05.112 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:33:05.112 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:33:05.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:33:05.415 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:05.416 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:05.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-fuse 2024-03-29T10:33:05.826 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:05.977 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:05.977 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:33:05.978 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-29T10:33:05.979 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-03-29T10:33:05.979 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:05.979 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 6.8 M 2024-03-29T10:33:05.979 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 33 M 2024-03-29T10:33:05.979 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:33:06.245 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:06.396 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:06.396 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:06.396 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:33:06.396 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:06.396 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:33:06.396 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:33:06.396 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:33:06.396 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:33:06.397 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:06.397 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:33:06.397 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:06.397 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-03-29T10:33:06.397 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:06.397 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 964 k 2024-03-29T10:33:06.397 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.8 M 2024-03-29T10:33:06.397 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:33:06.478 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-29T10:33:06.541 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 546 kB/s | 79 kB 00:00 2024-03-29T10:33:06.553 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T10:33:06.578 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 451 kB/s | 138 kB 00:00 2024-03-29T10:33:06.728 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 330 kB/s | 58 kB 00:00 2024-03-29T10:33:06.762 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 278 kB/s | 51 kB 00:00 2024-03-29T10:33:06.774 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 885 kB 00:00 2024-03-29T10:33:06.776 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:06.776 INFO:teuthology.orchestra.run.smithi117.stdout:Total 2.5 MB/s | 964 kB 00:00 2024-03-29T10:33:06.776 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:33:06.796 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:33:06.797 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:33:06.862 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-03-29T10:33:06.863 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:06.864 INFO:teuthology.orchestra.run.smithi136.stdout:Total 7.7 MB/s | 6.8 MB 00:00 2024-03-29T10:33:06.870 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:33:06.870 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:33:06.877 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:33:06.887 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:33:06.887 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:33:06.974 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:33:06.975 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:33:07.062 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:33:07.208 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:33:07.268 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:33:07.274 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:07.432 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:33:07.481 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:33:07.909 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:07.910 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:33:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:33:08.111 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:33:08.181 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:33:08.190 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:33:08.191 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:08.191 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:33:08.191 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:33:08.191 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:08.191 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:08.210 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:08.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-volume 2024-03-29T10:33:09.026 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:09.026 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:33:09.026 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:33:09.026 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:33:09.026 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:33:09.026 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:33:09.065 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:09.221 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:09.221 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:09.221 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T10:33:09.221 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:09.221 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:33:09.222 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:33:09.222 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T10:33:09.222 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:33:09.222 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:09.222 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:33:09.222 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:09.222 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-03-29T10:33:09.222 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:09.223 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 332 k 2024-03-29T10:33:09.223 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.5 M 2024-03-29T10:33:09.223 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:33:09.325 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:33:09.325 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:09.325 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:33:09.325 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:09.326 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:33:09.326 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:33:09.326 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:33:09.326 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:33:09.326 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:33:09.326 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:09.326 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:09.347 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 553 kB/s | 69 kB 00:00 2024-03-29T10:33:09.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-fuse 2024-03-29T10:33:09.622 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 660 kB/s | 264 kB 00:00 2024-03-29T10:33:09.623 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:09.624 INFO:teuthology.orchestra.run.smithi117.stdout:Total 828 kB/s | 332 kB 00:00 2024-03-29T10:33:09.624 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:33:09.632 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:33:09.632 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:33:09.667 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:33:09.668 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:33:09.813 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:33:10.041 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:10.057 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:10.057 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:10.057 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:33:10.058 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:10.137 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:10.288 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:10.289 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:10.289 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:33:10.289 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:10.289 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:33:10.289 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:33:10.289 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:33:10.289 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:33:10.289 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:10.290 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:33:10.290 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:10.290 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-03-29T10:33:10.290 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:10.290 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 964 k 2024-03-29T10:33:10.290 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.8 M 2024-03-29T10:33:10.291 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:33:10.423 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 595 kB/s | 79 kB 00:00 2024-03-29T10:33:10.606 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 885 kB 00:00 2024-03-29T10:33:10.607 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:10.608 INFO:teuthology.orchestra.run.smithi136.stdout:Total 3.0 MB/s | 964 kB 00:00 2024-03-29T10:33:10.608 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:33:10.628 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:33:10.628 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:33:10.701 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:33:10.701 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:33:10.774 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:33:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:33:11.040 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:33:11.107 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:33:11.107 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:11.107 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:33:11.107 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:11.107 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:33:11.107 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:11.107 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:11.118 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:11.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados-devel 2024-03-29T10:33:11.808 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:11.808 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:33:12.048 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:12.108 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:33:12.108 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:12.108 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:33:12.109 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:33:12.109 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:12.109 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:12.197 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:12.198 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 127 k 2024-03-29T10:33:12.199 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 456 k 2024-03-29T10:33:12.199 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:33:12.436 INFO:teuthology.orchestra.run.smithi117.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 534 kB/s | 127 kB 00:00 2024-03-29T10:33:12.437 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:12.437 INFO:teuthology.orchestra.run.smithi117.stdout:Total 530 kB/s | 127 kB 00:00 2024-03-29T10:33:12.437 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:33:12.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-volume 2024-03-29T10:33:12.442 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:33:12.442 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:33:12.501 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:33:12.502 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:33:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:33:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:13.026 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:13.179 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:13.180 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-29T10:33:13.180 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:13.180 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:33:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:33:13.181 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-29T10:33:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:33:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:13.181 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:33:13.181 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:13.181 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-03-29T10:33:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:13.182 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 332 k 2024-03-29T10:33:13.182 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 1.5 M 2024-03-29T10:33:13.182 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:33:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:13.295 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 601 kB/s | 69 kB 00:00 2024-03-29T10:33:13.429 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-29T10:33:13.430 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:13.431 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-29T10:33:13.431 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:33:13.439 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:33:13.439 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:33:13.477 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:33:13.477 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:33:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:13.489 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:33:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:13.489 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:13.625 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:33:13.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs2 2024-03-29T10:33:13.786 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:33:13.824 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:13.841 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:13.841 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:13.841 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:33:13.841 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:14.307 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:14.408 INFO:teuthology.orchestra.run.smithi117.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:14.463 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:14.464 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T10:33:14.464 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:14.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs-devel 2024-03-29T10:33:14.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:33:14.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:33:14.900 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:14.900 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:33:14.900 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:14.900 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:33:14.900 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:14.900 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:15.119 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:15.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados-devel 2024-03-29T10:33:15.270 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:15.271 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 31 k 2024-03-29T10:33:15.272 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 140 k 2024-03-29T10:33:15.272 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:33:15.448 INFO:teuthology.orchestra.run.smithi117.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 177 kB/s | 31 kB 00:00 2024-03-29T10:33:15.448 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:15.448 INFO:teuthology.orchestra.run.smithi117.stdout:Total 176 kB/s | 31 kB 00:00 2024-03-29T10:33:15.448 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:33:15.452 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:33:15.452 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:33:15.485 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:33:15.485 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:33:15.617 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:33:15.712 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:15.822 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:15.974 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:15.974 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 127 k 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 456 k 2024-03-29T10:33:15.975 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:33:16.066 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:16.294 INFO:teuthology.orchestra.run.smithi136.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 397 kB/s | 127 kB 00:00 2024-03-29T10:33:16.295 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:16.295 INFO:teuthology.orchestra.run.smithi136.stdout:Total 395 kB/s | 127 kB 00:00 2024-03-29T10:33:16.295 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:33:16.301 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:33:16.301 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:33:16.337 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:16.337 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:16.337 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:33:16.337 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:16.337 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:16.338 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:16.357 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:33:16.357 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:33:16.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados2 2024-03-29T10:33:16.513 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:33:16.618 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:17.060 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:17.157 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:17.208 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:17.209 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T10:33:17.209 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:17.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librbd1 2024-03-29T10:33:17.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:17.407 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:17.407 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:33:17.407 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:17.407 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:17.407 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:17.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs2 2024-03-29T10:33:17.814 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:17.911 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:17.963 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:17.964 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T10:33:17.964 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:18.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rados 2024-03-29T10:33:18.242 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:18.344 INFO:teuthology.orchestra.run.smithi136.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:18.399 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:18.400 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T10:33:18.400 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:18.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs-devel 2024-03-29T10:33:18.573 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:18.670 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:18.722 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:18.722 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T10:33:18.723 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:18.773 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rgw 2024-03-29T10:33:19.048 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:19.200 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:19.201 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-29T10:33:19.201 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 31 k 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 140 k 2024-03-29T10:33:19.202 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:33:19.339 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:19.438 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:19.448 INFO:teuthology.orchestra.run.smithi136.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 126 kB/s | 31 kB 00:00 2024-03-29T10:33:19.449 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:19.449 INFO:teuthology.orchestra.run.smithi136.stdout:Total 126 kB/s | 31 kB 00:00 2024-03-29T10:33:19.449 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:33:19.453 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:33:19.453 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:33:19.473 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:33:19.473 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:33:19.491 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:19.493 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T10:33:19.493 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:19.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-cephfs 2024-03-29T10:33:19.606 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:33:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:20.082 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:20.106 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:20.203 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:20.254 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:20.255 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T10:33:20.255 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:20.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rbd 2024-03-29T10:33:20.345 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:20.346 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:20.346 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:33:20.346 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:20.346 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:20.346 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:20.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados2 2024-03-29T10:33:20.877 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:20.976 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:21.028 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:21.029 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T10:33:21.029 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:21.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-fuse 2024-03-29T10:33:21.077 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:21.174 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:21.226 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:21.227 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T10:33:21.227 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:21.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librbd1 2024-03-29T10:33:21.643 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:21.795 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:21.796 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:21.796 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-03-29T10:33:21.796 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:21.796 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:33:21.796 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:33:21.796 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:21.797 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:33:21.797 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:21.797 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T10:33:21.797 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:21.797 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 85 k 2024-03-29T10:33:21.797 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 231 k 2024-03-29T10:33:21.797 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:33:21.842 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:21.939 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:21.991 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:21.992 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T10:33:21.992 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:22.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rados 2024-03-29T10:33:22.225 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 199 kB/s | 85 kB 00:00 2024-03-29T10:33:22.226 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:22.226 INFO:teuthology.orchestra.run.smithi117.stdout:Total 199 kB/s | 85 kB 00:00 2024-03-29T10:33:22.226 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:33:22.233 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:33:22.233 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:33:22.289 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:33:22.289 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:33:22.431 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:33:22.540 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:22.608 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:22.708 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:22.761 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:22.761 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T10:33:22.762 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:22.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rgw 2024-03-29T10:33:22.944 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:23.255 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:23.255 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:23.256 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:33:23.256 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:23.256 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:23.256 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:23.380 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:23.478 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:23.531 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:23.532 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T10:33:23.532 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:23.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-mirror 2024-03-29T10:33:23.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-cephfs 2024-03-29T10:33:24.133 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:24.153 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:24.251 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:24.291 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:24.292 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:24.292 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-29T10:33:24.292 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:24.292 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:33:24.292 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:33:24.292 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:24.292 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:33:24.292 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:24.293 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T10:33:24.293 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:24.293 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 3.1 M 2024-03-29T10:33:24.293 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 13 M 2024-03-29T10:33:24.293 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:33:24.302 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:24.303 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T10:33:24.303 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:24.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rbd 2024-03-29T10:33:24.838 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.7 MB/s | 3.1 MB 00:00 2024-03-29T10:33:24.838 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:24.839 INFO:teuthology.orchestra.run.smithi117.stdout:Total 5.7 MB/s | 3.1 MB 00:00 2024-03-29T10:33:24.839 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:33:24.848 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:33:24.848 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:33:24.917 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:24.921 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:33:24.921 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:33:25.014 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:25.066 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:25.067 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-29T10:33:25.067 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:25.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-fuse 2024-03-29T10:33:25.350 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:33:25.405 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:25.430 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:25.430 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:25.430 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:33:25.430 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:33:25.430 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:33:25.430 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:25.671 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:25.818 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:25.819 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:25.819 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-29T10:33:25.819 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:25.819 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:33:25.819 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:33:25.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:25.820 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:33:25.820 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:25.820 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T10:33:25.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:25.820 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 85 k 2024-03-29T10:33:25.820 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 231 k 2024-03-29T10:33:25.820 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:33:26.176 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 240 kB/s | 85 kB 00:00 2024-03-29T10:33:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:Total 239 kB/s | 85 kB 00:00 2024-03-29T10:33:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:33:26.184 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:33:26.184 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:33:26.238 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:33:26.239 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:33:26.375 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:33:26.474 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:26.619 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:26.620 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:26.620 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:33:26.620 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:26.620 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:26.620 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:26.844 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-nbd 2024-03-29T10:33:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:27.147 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:33:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:27.147 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:27.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-mirror 2024-03-29T10:33:27.418 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:31:34 AM UTC. 2024-03-29T10:33:27.566 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T10:33:27.567 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:27.567 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-03-29T10:33:27.567 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:27.567 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T10:33:27.567 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:33:27.567 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:27.567 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T10:33:27.567 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T10:33:27.568 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T10:33:27.568 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:27.568 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 171 k 2024-03-29T10:33:27.568 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 494 k 2024-03-29T10:33:27.568 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T10:33:27.908 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 505 kB/s | 171 kB 00:00 2024-03-29T10:33:27.908 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:27.908 INFO:teuthology.orchestra.run.smithi117.stdout:Total 503 kB/s | 171 kB 00:00 2024-03-29T10:33:27.909 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T10:33:27.916 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T10:33:27.916 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T10:33:27.973 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T10:33:27.973 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T10:33:27.992 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:28.124 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T10:33:28.148 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:28.148 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.1 M 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 13 M 2024-03-29T10:33:28.149 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:33:28.240 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:28.670 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:28.714 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.5 MB/s | 3.1 MB 00:00 2024-03-29T10:33:28.715 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:28.716 INFO:teuthology.orchestra.run.smithi136.stdout:Total 5.4 MB/s | 3.1 MB 00:00 2024-03-29T10:33:28.716 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:33:28.725 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:33:28.725 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:33:28.797 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:33:28.798 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:33:28.968 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:28.968 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:28.968 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T10:33:28.968 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:28.968 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:28.968 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T10:33:29.219 DEBUG:teuthology.parallel:result is None 2024-03-29T10:33:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:33:29.290 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:29.315 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:29.315 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:29.315 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:33:29.315 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:33:29.315 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:33:29.315 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:30.521 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:33:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:30.522 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:30.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-nbd 2024-03-29T10:33:31.357 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 10:31:35 AM UTC. 2024-03-29T10:33:31.505 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-29T10:33:31.506 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:31.507 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 171 k 2024-03-29T10:33:31.507 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 494 k 2024-03-29T10:33:31.507 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-29T10:33:31.861 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 486 kB/s | 171 kB 00:00 2024-03-29T10:33:31.861 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:31.861 INFO:teuthology.orchestra.run.smithi136.stdout:Total 484 kB/s | 171 kB 00:00 2024-03-29T10:33:31.861 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-29T10:33:31.868 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-29T10:33:31.869 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-29T10:33:31.924 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-29T10:33:31.924 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-29T10:33:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-29T10:33:32.175 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:32.877 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-29T10:33:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:32.878 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-29T10:33:33.122 DEBUG:teuthology.parallel:result is None 2024-03-29T10:33:33.122 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:33:33.342 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:33:33.367 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:33:33.368 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:33:33.368 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:33:33.370 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:33:33.588 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:33:33.610 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:33:33.610 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:33:33.610 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:33:33.612 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:33:33.612 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:33.613 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:33:33.645 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:33:33.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:33:33.678 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:33:33.678 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:33.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:33:33.715 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:33:33.786 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:33:33.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:33:33.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:33:33.880 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:33:33.880 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:33.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:33:33.909 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:33:33.977 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:33:33.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:33:34.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:33:34.070 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:33:34.070 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:34.071 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:33:34.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:33:34.164 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:33:34.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:33:34.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:33:34.259 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:33:34.520 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:33:34.521 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2024-03-29T10:33:34.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2024-03-29T10:33:34.551 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:33:34.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:33:34.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:33:34.618 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:33:34.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:33:34.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:33:34.696 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:33:34.696 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:33:34.696 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622640', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJuXhtiIzox6RyjMmMKLrraT+wv1+0A0OvY4n+MklX/S7b31OH0r/OKQJhnGds7aodC+WjKO8CpeIt5U8dLrpp0=', 'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBk5LqafpU9UWpW47/yw6pdgu4pgdDLnxkVKd1NImpWtyodgt+PmSbZZ0nRc07yD+cTa1jWNYT9NCp59uKxCtw0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:33:34.696 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:33:34.726 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:33:34.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:33:34.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:33:34.807 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:34.807 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:33:34.861 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:33:34.861 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-03-29T10:33:34.916 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:33:34.917 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:33:34.917 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T10:33:34.917 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:33:34.917 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:33:34.917 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-29 10:33:25.703619678 +0000 2024-03-29T10:33:34.917 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-29 10:30:50.816554971 +0000 2024-03-29T10:33:34.918 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-29 10:30:50.816554971 +0000 2024-03-29T10:33:34.918 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-29 10:30:50.816554971 +0000 2024-03-29T10:33:34.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:33:34.982 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-29T10:33:34.982 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-29T10:33:34.982 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.00021578 s, 2.4 MB/s 2024-03-29T10:33:34.983 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:33:35.041 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-03-29T10:33:35.097 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:33:35.097 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:33:35.097 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T10:33:35.097 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:33:35.097 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:33:35.097 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-29 10:33:25.703619678 +0000 2024-03-29T10:33:35.097 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-29 10:30:51.178547897 +0000 2024-03-29T10:33:35.097 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-29 10:30:51.178547897 +0000 2024-03-29T10:33:35.097 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-29 10:30:51.178547897 +0000 2024-03-29T10:33:35.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:33:35.163 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-29T10:33:35.163 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-29T10:33:35.163 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000205094 s, 2.5 MB/s 2024-03-29T10:33:35.165 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:33:35.221 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-03-29T10:33:35.276 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:33:35.276 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:33:35.276 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T10:33:35.276 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:33:35.276 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:33:35.276 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-29 10:33:25.704619659 +0000 2024-03-29T10:33:35.276 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-29 10:30:51.526541096 +0000 2024-03-29T10:33:35.276 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-29 10:30:51.526541096 +0000 2024-03-29T10:33:35.277 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-29 10:30:51.526541096 +0000 2024-03-29T10:33:35.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:33:35.341 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-29T10:33:35.341 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-29T10:33:35.341 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000192709 s, 2.7 MB/s 2024-03-29T10:33:35.342 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:33:35.399 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-03-29T10:33:35.454 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:33:35.454 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:33:35.454 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T10:33:35.454 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:33:35.454 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:33:35.454 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-29 10:33:25.704619659 +0000 2024-03-29T10:33:35.454 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-29 10:30:51.843534901 +0000 2024-03-29T10:33:35.454 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-29 10:30:51.843534901 +0000 2024-03-29T10:33:35.454 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-29 10:30:51.843534901 +0000 2024-03-29T10:33:35.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:33:35.519 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-29T10:33:35.519 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-29T10:33:35.519 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000183924 s, 2.8 MB/s 2024-03-29T10:33:35.521 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:33:35.576 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:35.577 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:33:35.577 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:33:35.594 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:33:35.594 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-03-29T10:33:35.650 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:33:35.650 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:33:35.650 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-29T10:33:35.651 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:33:35.651 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:33:35.651 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-29 10:33:29.598331201 +0000 2024-03-29T10:33:35.651 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-29 10:30:50.809602246 +0000 2024-03-29T10:33:35.651 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-29 10:30:50.809602246 +0000 2024-03-29T10:33:35.651 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-29 10:30:50.809602246 +0000 2024-03-29T10:33:35.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:33:35.718 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-29T10:33:35.718 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-29T10:33:35.718 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000189831 s, 2.7 MB/s 2024-03-29T10:33:35.719 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:33:35.776 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-03-29T10:33:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:33:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:33:35.832 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-29T10:33:35.832 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:33:35.832 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:33:35.832 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-29 10:33:29.598331201 +0000 2024-03-29T10:33:35.833 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-29 10:30:51.167594370 +0000 2024-03-29T10:33:35.833 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-29 10:30:51.167594370 +0000 2024-03-29T10:33:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-29 10:30:51.167594370 +0000 2024-03-29T10:33:35.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:33:35.899 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-29T10:33:35.900 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-29T10:33:35.900 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000294143 s, 1.7 MB/s 2024-03-29T10:33:35.901 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:33:35.957 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-03-29T10:33:36.012 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:33:36.012 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:33:36.012 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-29T10:33:36.012 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:33:36.012 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:33:36.012 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-29 10:33:29.599331179 +0000 2024-03-29T10:33:36.013 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-29 10:30:51.500587043 +0000 2024-03-29T10:33:36.013 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-29 10:30:51.500587043 +0000 2024-03-29T10:33:36.013 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-29 10:30:51.500587043 +0000 2024-03-29T10:33:36.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:33:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-29T10:33:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-29T10:33:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000231678 s, 2.2 MB/s 2024-03-29T10:33:36.080 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:33:36.137 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-03-29T10:33:36.192 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:33:36.192 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:33:36.192 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-29T10:33:36.192 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:33:36.192 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:33:36.192 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-29 10:33:29.599331179 +0000 2024-03-29T10:33:36.192 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-29 10:30:51.838579606 +0000 2024-03-29T10:33:36.192 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-29 10:30:51.838579606 +0000 2024-03-29T10:33:36.193 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-29 10:30:51.838579606 +0000 2024-03-29T10:33:36.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:33:36.258 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-29T10:33:36.258 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-29T10:33:36.258 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000212259 s, 2.4 MB/s 2024-03-29T10:33:36.259 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:33:36.314 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:36.314 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:33:36.314 INFO:tasks.ceph:Generating config... 2024-03-29T10:33:36.329 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:33:36.329 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:33:36.329 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:33:36.329 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:33:36.329 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:33:36.329 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:33:36.329 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:33:36.329 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:33:36.330 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:33:36.330 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:33:36.330 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:33:36.330 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:33:36.331 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:33:36.332 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:33:36.332 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:33:36.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:33:36.398 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:33:36.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:33:36.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:33:36.476 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.117:6789]'), ('mon.c', '[v1:172.21.15.117:6790]'), ('mon.b', '[v1:172.21.15.136:6789]')] 2024-03-29T10:33:36.476 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.117:6789],[v1:172.21.15.117:6790],[v1:172.21.15.136:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:33:36.477 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:36.477 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:33:36.532 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.117:6789]' --addv c '[v1:172.21.15.117:6790]' --addv b '[v1:172.21.15.136:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:36.619 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:36.619 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: generated fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:33:36.619 INFO:teuthology.orchestra.run.smithi117.stdout:setting min_mon_release = quincy 2024-03-29T10:33:36.619 INFO:teuthology.orchestra.run.smithi117.stdout:epoch 0 2024-03-29T10:33:36.619 INFO:teuthology.orchestra.run.smithi117.stdout:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:33:36.619 INFO:teuthology.orchestra.run.smithi117.stdout:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:33:36.619 INFO:teuthology.orchestra.run.smithi117.stdout:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:33:36.619 INFO:teuthology.orchestra.run.smithi117.stdout:min_mon_release 17 (quincy) 2024-03-29T10:33:36.620 INFO:teuthology.orchestra.run.smithi117.stdout:election_strategy: 1 2024-03-29T10:33:36.620 INFO:teuthology.orchestra.run.smithi117.stdout:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:33:36.620 INFO:teuthology.orchestra.run.smithi117.stdout:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:33:36.620 INFO:teuthology.orchestra.run.smithi117.stdout:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:33:36.620 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:33:36.621 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:33:36.636 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 539ca054-22ca-4fb7-a22c-88dad95af2f1... 2024-03-29T10:33:36.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:33:36.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:33:36.726 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-03-29T10:33:36.726 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-03-29T10:33:36.727 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:33:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-03-29T10:33:36.729 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "[v1:172.21.15.117:6789],[v1:172.21.15.117:6790],[v1:172.21.15.136:6789]" 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: lockdep = True 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: mon client directed command retry = 5 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: mon election default strategy = 1 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: ms bind msgr2 = False 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: ms inject socket failures = 5000 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: ms type = async 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-03-29T10:33:36.730 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug trim objects = true 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:33:36.731 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore block size = 96636764160 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore compression algorithm = zstd 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore compression mode = aggressive 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore fsck on mount = True 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore zero block detection = True 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 20 2024-03-29T10:33:36.732 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 20 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 10 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd full ratio = 0.9 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: osd class default list = * 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: osd class load list = * 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: osd client watch timeout = 120 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify cached snaps = True 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify missing on start = True 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock override recovery settings = True 2024-03-29T10:33:36.733 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = bluestore 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue = debug_random 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue cut off = debug_random 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: osd shutdown pgref assert = True 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:33:36.734 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:33:36.735 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: mon min osdmap epochs = 50 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune interval = 2 2024-03-29T10:33:36.736 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune min = 15 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: mon scrub interval = 300 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on pool no app = False 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: paxos service trim min = 10 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-29T10:33:36.737 INFO:teuthology.orchestra.run.smithi117.stdout: debug objecter = 20 2024-03-29T10:33:36.738 INFO:teuthology.orchestra.run.smithi117.stdout: debug rados = 20 2024-03-29T10:33:36.738 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-03-29T10:33:36.738 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2024-03-29T10:33:36.738 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2024-03-29T10:33:36.738 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-03-29T10:33:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-03-29T10:33:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:33:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-03-29T10:33:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:33:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:33:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-03-29T10:33:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:33:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-03-29T10:33:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-03-29T10:33:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = "[v1:172.21.15.117:6789],[v1:172.21.15.117:6790],[v1:172.21.15.136:6789]" 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: lockdep = True 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: mon election default strategy = 1 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: ms bind msgr2 = False 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 5000 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: ms type = async 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-03-29T10:33:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-03-29T10:33:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug trim objects = true 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-03-29T10:33:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore compression algorithm = zstd 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore compression mode = aggressive 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore zero block detection = True 2024-03-29T10:33:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 20 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 20 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: debug monc = 20 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 10 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: osd class default list = * 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: osd class load list = * 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: osd client watch timeout = 120 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify cached snaps = True 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify missing on start = True 2024-03-29T10:33:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:33:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock override recovery settings = True 2024-03-29T10:33:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2024-03-29T10:33:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue = debug_random 2024-03-29T10:33:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue cut off = debug_random 2024-03-29T10:33:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: osd shutdown pgref assert = True 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.748 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:33:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: mon min osdmap epochs = 50 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune interval = 2 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune min = 15 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: mon scrub interval = 300 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on pool no app = False 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: paxos service trim min = 10 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:33:36.751 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:33:36.752 INFO:teuthology.orchestra.run.smithi136.stdout: debug monc = 20 2024-03-29T10:33:36.752 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-29T10:33:36.752 INFO:teuthology.orchestra.run.smithi136.stdout: debug objecter = 20 2024-03-29T10:33:36.752 INFO:teuthology.orchestra.run.smithi136.stdout: debug rados = 20 2024-03-29T10:33:36.752 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-03-29T10:33:36.752 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.c] 2024-03-29T10:33:36.752 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.b] 2024-03-29T10:33:36.754 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:33:36.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:33:36.850 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:33:36.850 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:36.851 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:33:36.865 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:36.865 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:33:36.920 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:33:36.921 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:36.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:33:36.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:33:37.001 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:33:37.001 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:37.056 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:33:37.056 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:33:37.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:33:37.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:33:37.096 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:33:37.096 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:37.151 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:33:37.151 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:33:37.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:33:37.205 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:33:37.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:33:37.269 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:33:37.271 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:33:37.271 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:33:37.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:33:37.314 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:33:37.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:33:37.371 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:33:37.386 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:33:37.386 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:33:37.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:33:37.413 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:37.414 INFO:tasks.ceph:role: osd.0 2024-03-29T10:33:37.414 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:33:37.414 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:33:37.487 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:33:37.487 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:33:37.487 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:33:37.487 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:33:37.487 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:33:37.487 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:33:37.487 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:33:37.487 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:33:37.487 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:33:37.488 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:33:37.992 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-29T10:33:37.994 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-29T10:33:37.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:33:38.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:33:38.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:33:38.249 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:38.249 INFO:tasks.ceph:role: osd.1 2024-03-29T10:33:38.249 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:33:38.249 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:33:38.324 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:33:38.325 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:33:38.325 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:33:38.325 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:33:38.325 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:33:38.325 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:33:38.325 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:33:38.325 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:33:38.325 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:33:38.325 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:33:38.770 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-29T10:33:38.771 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-29T10:33:38.772 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:33:38.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:33:38.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:33:38.975 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:38.976 INFO:tasks.ceph:role: osd.2 2024-03-29T10:33:38.976 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:33:38.976 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:33:39.050 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:33:39.050 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:33:39.050 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:33:39.050 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:33:39.050 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:33:39.050 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:33:39.050 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:33:39.050 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:33:39.050 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:33:39.051 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:33:39.539 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-29T10:33:39.541 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-29T10:33:39.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:33:39.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:33:39.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:33:39.791 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:39.791 INFO:tasks.ceph:role: osd.3 2024-03-29T10:33:39.791 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:33:39.791 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:33:39.866 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:33:39.866 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:33:39.866 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:33:39.866 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:33:39.866 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:33:39.866 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:33:39.866 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:33:39.866 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:33:39.866 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:33:39.867 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:33:40.376 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-29T10:33:40.378 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-29T10:33:40.378 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:33:40.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:33:40.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:40.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:40.670+0000 7f86d6f6d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:33:40.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:40.670+0000 7f86d6f6d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:33:40.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:40.670+0000 7f86d6f6d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:33:40.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:40.670+0000 7f86d6f6d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:33:40.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:40.670+0000 7f86d6f6d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:33:40.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:40.670+0000 7f86d6f6d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:33:43.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:33:43.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:43.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:43.647+0000 7f846a6ae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:33:43.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:43.647+0000 7f846a6ae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:33:43.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:43.647+0000 7f846a6ae640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:33:43.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:43.647+0000 7f846a6ae640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:33:43.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:43.647+0000 7f846a6ae640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:33:43.634 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:43.648+0000 7f846a6ae640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:33:46.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:33:46.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:46.617+0000 7f1bee013640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:33:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:46.617+0000 7f1bee013640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:33:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:46.617+0000 7f1bee013640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:33:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:46.617+0000 7f1bee013640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:33:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:46.617+0000 7f1bee013640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:33:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:46.618+0000 7f1bee013640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:33:49.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:33:49.492 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:49.581 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:49.595+0000 7f974ca89640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:33:49.581 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:49.596+0000 7f974ca89640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:33:49.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:49.596+0000 7f974ca89640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:33:49.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:49.596+0000 7f974ca89640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:33:49.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:49.596+0000 7f974ca89640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:33:49.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:33:49.596+0000 7f974ca89640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:33:52.440 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:33:52.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:33:52.504 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:52.504 INFO:tasks.ceph:role: osd.4 2024-03-29T10:33:52.504 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:33:52.504 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:33:52.583 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:33:52.583 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:33:52.583 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:33:52.584 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:33:52.584 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:33:52.584 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:33:52.584 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:33:52.584 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:33:52.584 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:33:52.584 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:33:53.103 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-29T10:33:53.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-29T10:33:53.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:33:53.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:33:53.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:33:53.318 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:53.318 INFO:tasks.ceph:role: osd.5 2024-03-29T10:33:53.318 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:33:53.318 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:33:53.393 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:33:53.393 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:33:53.393 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:33:53.394 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:33:53.394 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:33:53.394 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:33:53.394 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:33:53.394 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:33:53.394 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:33:53.394 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:33:53.894 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-29T10:33:53.897 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-29T10:33:53.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:33:53.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:33:54.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:33:54.100 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:54.100 INFO:tasks.ceph:role: osd.6 2024-03-29T10:33:54.100 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:33:54.100 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:33:54.175 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:33:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:33:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:33:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:33:54.175 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:33:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:33:54.176 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:33:54.176 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:33:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:33:54.176 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:33:54.687 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-29T10:33:54.689 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-29T10:33:54.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:33:54.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:33:54.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:33:54.897 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:33:54.898 INFO:tasks.ceph:role: osd.7 2024-03-29T10:33:54.898 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:33:54.898 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:33:54.973 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:33:55.497 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-29T10:33:55.499 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-29T10:33:55.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:33:55.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:33:55.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:55.777+0000 7f56dbc48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:33:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:55.778+0000 7f56dbc48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:33:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:55.778+0000 7f56dbc48640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:33:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:55.778+0000 7f56dbc48640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:33:55.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:55.778+0000 7f56dbc48640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:33:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:55.782+0000 7f56dbc48640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:33:58.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:33:58.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:33:58.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:58.757+0000 7f417fd11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:33:58.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:58.757+0000 7f417fd11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:33:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:58.757+0000 7f417fd11640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:33:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:58.757+0000 7f417fd11640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:33:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:58.757+0000 7f417fd11640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:33:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:33:58.758+0000 7f417fd11640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:34:01.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:34:01.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:01.735+0000 7f00a8415640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:34:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:01.735+0000 7f00a8415640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:34:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:01.735+0000 7f00a8415640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:34:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:01.735+0000 7f00a8415640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:34:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:01.735+0000 7f00a8415640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:34:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:01.736+0000 7f00a8415640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:34:04.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:34:04.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:04.706+0000 7fdcd600d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:34:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:04.706+0000 7fdcd600d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:34:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:04.706+0000 7fdcd600d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:34:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:04.706+0000 7fdcd600d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:34:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:04.706+0000 7fdcd600d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:34:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:04.707+0000 7fdcd600d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:34:07.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:34:07.567 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:34:07.567 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:07.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:34:07.599 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:07.599 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:34:07.670 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:07.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:34:07.739 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:07.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:34:07.807 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:07.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:34:07.876 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:07.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:34:07.908 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:07.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:34:07.979 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:07.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:34:08.050 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:08.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:34:08.119 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:08.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:34:08.188 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:08.188 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:34:08.204 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:08.204 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:34:08.244 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:34:08.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:34:08.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:34:08.287 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.y] 2024-03-29T10:34:08.287 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQABmQZm0nYsDRAAxMk3LBvzv9lKQWa7r0VHjA== 2024-03-29T10:34:08.287 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAEmQZmwXb+JxAAMykGRya0+Ghol7DTCwex1w== 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAHmQZmw0elJhAAU+ffo2VgG+MCDwrwH8ZOSw== 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAKmQZm9wHYJBAAEjRDxK5aPXgxLjjgPN22zg== 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.3] 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQANmQZmV+SQIxAAp9FEBEa+XubnygGP9RnNrA== 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQABmQZm/nf7EBAAH0QzjpUjB1Be4ThVk0wXbQ== 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.4] 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQATmQZmNB1rLhAAJ+hoPwe89jyPihs8DIhJsg== 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.5] 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAWmQZmuwcwLRAApKSNxPysbw/1PokqlXFHTA== 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.6] 2024-03-29T10:34:08.288 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAZmQZmFyTmKxAAR95TIcimX687IjyUOWzYwA== 2024-03-29T10:34:08.289 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.7] 2024-03-29T10:34:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAcmQZmHF4mKhAAfoj5n9Ltk/y0oSJWkMRFTQ== 2024-03-29T10:34:08.289 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-03-29T10:34:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQABmQZmkm6pExAAYx++bhXmVp6Wsa7+7HD0Mw== 2024-03-29T10:34:08.289 INFO:teuthology.orchestra.run.smithi117.stdout:[client.1] 2024-03-29T10:34:08.289 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQABmQZm5sgQFxAAUoKDMm7ORSkCngES84I9vw== 2024-03-29T10:34:08.310 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.y] 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQABmQZm0nYsDRAAxMk3LBvzv9lKQWa7r0VHjA== 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQAEmQZmwXb+JxAAMykGRya0+Ghol7DTCwex1w== 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQAHmQZmw0elJhAAU+ffo2VgG+MCDwrwH8ZOSw== 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQAKmQZm9wHYJBAAEjRDxK5aPXgxLjjgPN22zg== 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.3] 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQANmQZmV+SQIxAAp9FEBEa+XubnygGP9RnNrA== 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQABmQZm/nf7EBAAH0QzjpUjB1Be4ThVk0wXbQ== 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.4] 2024-03-29T10:34:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQATmQZmNB1rLhAAJ+hoPwe89jyPihs8DIhJsg== 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.5] 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQAWmQZmuwcwLRAApKSNxPysbw/1PokqlXFHTA== 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.6] 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQAZmQZmFyTmKxAAR95TIcimX687IjyUOWzYwA== 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.7] 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQAcmQZmHF4mKhAAfoj5n9Ltk/y0oSJWkMRFTQ== 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQABmQZmkm6pExAAYx++bhXmVp6Wsa7+7HD0Mw== 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout:[client.1] 2024-03-29T10:34:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQABmQZm5sgQFxAAUoKDMm7ORSkCngES84I9vw== 2024-03-29T10:34:08.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:34:08.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:34:08.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.460 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.621 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:34:08.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:34:08.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:08.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:34:08.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:34:08.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:34:08.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:34:09.005 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:34:09.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:34:09.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:34:09.128 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:34:09.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:34:09.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:34:09.319 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:34:09.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:34:09.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:34:09.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:34:09.499 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:09.502 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:09.554 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:34:09.554 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:34:09.554 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:34:09.556 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:34:09.557 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:34:09.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:34:09.558 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:34:09.559 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:34:09.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:34:09.596 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:34:09.597 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:34:09.597 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:34:09.597 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:34:09.599 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:34:09.600 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:34:09.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:34:09.602 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:34:09.602 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:34:09.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:34:09.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.717+0000 7fe5da831640 10 monclient: get_monmap_and_config 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.717+0000 7fe5da831640 10 monclient: build_initial_monmap 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.717+0000 7fe5da831640 10 monclient: monmap: 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:34:09.718426+0000 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:34:09.718426+0000 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:09.703 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:09.704 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:09.704 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:09.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 1 Processor -- start 2024-03-29T10:34:09.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 1 -- start start 2024-03-29T10:34:09.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 10 monclient: init 2024-03-29T10:34:09.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:09.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:34:09.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 10 monclient(hunting): picked mon.noname-a con 0x7fe5d409ac40 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:09.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 10 monclient(hunting): picked mon.noname-b con 0x7fe5d409c870 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:09.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 10 monclient(hunting): picked mon.noname-c con 0x7fe5d40b4360 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:09.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d409dec0 con 0x7fe5d409ac40 2024-03-29T10:34:09.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d409da30 con 0x7fe5d40b4360 2024-03-29T10:34:09.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d40b39c0 con 0x7fe5d409c870 2024-03-29T10:34:09.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.718+0000 7fe5da831640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:09.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.719+0000 7fe5da831640 10 monclient(hunting): authenticate will time out at 770.981121s 2024-03-29T10:34:09.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.719+0000 7fe5d37fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe5d409c870 0x7fe5d413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34072/0 (socket says 172.21.15.117:34072) 2024-03-29T10:34:09.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.719+0000 7fe5d3fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe5d409ac40 0x7fe5d40b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:43714/0 (socket says 172.21.15.117:43714) 2024-03-29T10:34:09.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:09.719+0000 7fe5d37fe640 1 -- 172.21.15.117:0/1983283266 learned_addr learned my addr 172.21.15.117:0/1983283266 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 10 monclient(hunting): tick 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 1 monclient(hunting): continuing hunt 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 1 -- 172.21.15.117:0/1983283266 >> v1:172.21.15.117:6790/0 conn(0x7fe5d409c870 legacy=0x7fe5d413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 1 -- 172.21.15.117:0/1983283266 >> v1:172.21.15.136:6789/0 conn(0x7fe5d40b4360 legacy=0x7fe5d414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 1 -- 172.21.15.117:0/1983283266 >> v1:172.21.15.117:6789/0 conn(0x7fe5d409ac40 legacy=0x7fe5d40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c00036e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe5c00072b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:12.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe5c000afb0 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:12.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 1 -- 172.21.15.117:0/1983283266 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d40b5160 con 0x7fe5c00072b0 2024-03-29T10:34:12.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 1 -- 172.21.15.117:0/1983283266 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c000fc80 con 0x7fe5c000afb0 2024-03-29T10:34:12.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 1 -- 172.21.15.117:0/1983283266 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c0010ee0 con 0x7fe5c00036e0 2024-03-29T10:34:12.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:12.719+0000 7fe5d1ffb640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:14.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.884+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136361564 0 0) 0x7fe5d40b5160 con 0x7fe5c00072b0 2024-03-29T10:34:14.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.884+0000 7fe5d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.884+0000 7fe5d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.884+0000 7fe5d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.884+0000 7fe5d2ffd640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.884+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5bc003520 con 0x7fe5c00072b0 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213846038 0 0) 0x7fe5c0010ee0 con 0x7fe5c00036e0 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d40b5160 con 0x7fe5c00036e0 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4025227931 0 0) 0x7fe5d40b5160 con 0x7fe5c00036e0 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 >> v1:172.21.15.136:6789/0 conn(0x7fe5c000afb0 legacy=0x7fe5c000d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:14.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 >> v1:172.21.15.117:6789/0 conn(0x7fe5c00072b0 legacy=0x7fe5c0009720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 monclient: found mon.noname-b 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c0012140 con 0x7fe5c00036e0 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe5c40014f0 con 0x7fe5c00036e0 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5da831640 5 monclient: authenticate success, global_id 4106 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5da831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: dump: 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5da831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5c40016c0 con 0x7fe5c00036e0 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 1 -- 172.21.15.117:0/1983283266 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe5c4001a50 con 0x7fe5c00036e0 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5da831640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5da831640 4 set_mon_vals no callback set 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.885+0000 7fe5d2ffd640 10 monclient: dump: 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:14.875 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:14.876 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.886+0000 7fe5da831640 10 monclient: shutdown 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.887+0000 7fe5da831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.887+0000 7fe5da831640 1 -- 172.21.15.117:0/1983283266 >> v1:172.21.15.117:6790/0 conn(0x7fe5c00036e0 legacy=0x7fe5c0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.887+0000 7fe5da831640 1 -- 172.21.15.117:0/1983283266 shutdown_connections 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.887+0000 7fe5da831640 1 -- 172.21.15.117:0/1983283266 >> 172.21.15.117:0/1983283266 conn(0x7fe5d4066b00 msgr2=0x7fe5d4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.887+0000 7fe5da831640 1 -- 172.21.15.117:0/1983283266 shutdown_connections 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.887+0000 7fe5da831640 1 -- 172.21.15.117:0/1983283266 wait complete. 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.888+0000 7fe5da831640 10 monclient: build_initial_monmap 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.888+0000 7fe5da831640 10 monclient: monmap: 2024-03-29T10:34:14.881 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.888+0000 7fe5da831640 1 librados: starting msgr at 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.888+0000 7fe5da831640 1 librados: starting objecter 2024-03-29T10:34:14.882 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.888+0000 7fe5da831640 1 Processor -- start 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 1 -- start start 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 1 librados: setting wanted keys 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 1 librados: calling monclient init 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 10 monclient: init 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 10 monclient(hunting): picked mon.noname-a con 0x7fe5c00036e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 10 monclient(hunting): picked mon.noname-c con 0x7fe5c00072b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c000afb0 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d413a7f0 con 0x7fe5c00036e0 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5da831640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d4141e80 con 0x7fe5c000afb0 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5d3fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe5c00036e0 0x7fe5d413ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:51516/0 (socket says 172.21.15.117:51516) 2024-03-29T10:34:14.883 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.889+0000 7fe5d3fff640 1 -- 172.21.15.117:0/1977210827 learned_addr learned my addr 172.21.15.117:0/1977210827 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d4142640 con 0x7fe5c00072b0 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5da831640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5da831640 10 monclient(hunting): authenticate will time out at 776.152161s 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 client.?.objecter ms_handle_connect 0x7fe5c00036e0 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 client.?.objecter ms_handle_connect 0x7fe5c000afb0 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 client.?.objecter ms_handle_connect 0x7fe5c00072b0 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742807144 0 0) 0x7fe5d413a7f0 con 0x7fe5c00036e0 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:34:14.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c00105d0 con 0x7fe5c00036e0 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2731615681 0 0) 0x7fe5c00105d0 con 0x7fe5c00036e0 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5d413a7f0 con 0x7fe5c00036e0 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe5c4001430 con 0x7fe5c00036e0 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:14.885 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.890+0000 7fe5d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.891+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921793288 0 0) 0x7fe5d4141e80 con 0x7fe5c000afb0 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.891+0000 7fe5d0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.891+0000 7fe5d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.891+0000 7fe5d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.891+0000 7fe5d0ff9640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.891+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c00105d0 con 0x7fe5c000afb0 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183040530 0 0) 0x7fe5d4142640 con 0x7fe5c00072b0 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:34:14.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d4141e80 con 0x7fe5c00072b0 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2660684129 0 0) 0x7fe5c00105d0 con 0x7fe5c000afb0 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5d4142640 con 0x7fe5c000afb0 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe5cc001a10 con 0x7fe5c000afb0 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:14.887 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1506132320 0 0) 0x7fe5d413a7f0 con 0x7fe5c00036e0 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 >> v1:172.21.15.117:6790/0 conn(0x7fe5c00072b0 legacy=0x7fe5d41399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 >> v1:172.21.15.136:6789/0 conn(0x7fe5c000afb0 legacy=0x7fe5d413a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 monclient: found mon.a 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d40b1a70 con 0x7fe5c00036e0 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:14.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5da831640 5 monclient: authenticate success, global_id 4113 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5da831640 10 monclient: _renew_subs 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5da831640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d41422a0 con 0x7fe5c00036e0 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5da831640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5da831640 10 monclient: _renew_subs 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5da831640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d40b11b0 con 0x7fe5c00036e0 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.892+0000 7fe5da831640 1 librados: init done 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5da831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5da831640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5da831640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d4142490 con 0x7fe5c00036e0 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5c4004a90 con 0x7fe5c00036e0 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:14.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe5c4004da0 con 0x7fe5c00036e0 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 10 monclient: dump: 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:14.890 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:14.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d27fc640 4 set_mon_vals no callback set 2024-03-29T10:34:14.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe5c4005290 con 0x7fe5c00036e0 2024-03-29T10:34:14.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2073911830 0 0) 0x7fe5c4006780 con 0x7fe5c00036e0 2024-03-29T10:34:14.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 10 client.4113.objecter ms_dispatch 0x7fe5d4000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:34:14.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:34:14.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:34:14.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.894+0000 7fe5d0ff9640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T10:34:14.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.898+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe5c4006b10 con 0x7fe5c00036e0 2024-03-29T10:34:14.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.898+0000 7fe5d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:14.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.898+0000 7fe5d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:14.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.969+0000 7fe5da831640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:34:14.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.969+0000 7fe5da831640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:34:14.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.969+0000 7fe5da831640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:14.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.969+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe5d409bbb0 con 0x7fe5c00036e0 2024-03-29T10:34:14.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.971+0000 7fe5d0ff9640 1 -- 172.21.15.117:0/1977210827 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fe5c4005620 con 0x7fe5c00036e0 2024-03-29T10:34:14.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.971+0000 7fe5d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:34:14.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.971+0000 7fe5d0ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:34:14.957 INFO:teuthology.orchestra.run.smithi117.stderr:adjusted tunables profile to default 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 10 librados: watch_flush enter 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 10 librados: watch_flush exit 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 10 client.4113.objecter successfully canceled tick 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 10 monclient: shutdown 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 >> v1:172.21.15.117:6789/0 conn(0x7fe5c00036e0 legacy=0x7fe5d413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 shutdown_connections 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 >> 172.21.15.117:0/1977210827 conn(0x7fe5d4066b00 msgr2=0x7fe5d413af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:14.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 shutdown_connections 2024-03-29T10:34:14.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.972+0000 7fe5da831640 1 -- 172.21.15.117:0/1977210827 wait complete. 2024-03-29T10:34:14.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:14.973+0000 7fe5da831640 1 librados: shutdown 2024-03-29T10:34:14.969 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:34:14.969 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:34:14.969 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:14.969 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:34:14.996 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:14.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:34:15.063 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:15.063 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:34:15.130 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T10:34:15.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:34:15.197 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:15.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:34:15.224 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:15.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:34:15.291 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:15.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:34:15.358 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-29T10:34:15.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:34:15.378 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:15.392+0000 7f2e34034040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:34:15.382 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:15.396+0000 7f414a1ed040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:34:15.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new af857458-18ba-48bb-ba1c-5577c5a7ff12 0 2024-03-29T10:34:15.435 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:15.449+0000 7f2e34034040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:34:15.435 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:15.449+0000 7f414a1ed040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:34:15.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.549+0000 7f0421c63640 10 monclient: get_monmap_and_config 2024-03-29T10:34:15.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.549+0000 7f0421c63640 10 monclient: build_initial_monmap 2024-03-29T10:34:15.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.549+0000 7f0421c63640 10 monclient: monmap: 2024-03-29T10:34:15.534 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:15.535 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.535 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:34:15.549899+0000 2024-03-29T10:34:15.535 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:34:15.549899+0000 2024-03-29T10:34:15.535 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:15.535 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.535 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:15.535 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:15.535 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:15.535 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.549+0000 7f0421c63640 1 Processor -- start 2024-03-29T10:34:15.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 1 -- start start 2024-03-29T10:34:15.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 10 monclient: init 2024-03-29T10:34:15.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 10 monclient(hunting): picked mon.noname-a con 0x7f041c09ac40 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 10 monclient(hunting): picked mon.noname-b con 0x7f041c09c870 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 10 monclient(hunting): picked mon.noname-c con 0x7f041c0b4360 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041c09dec0 con 0x7f041c09ac40 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041c09da30 con 0x7f041c0b4360 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041c0b39c0 con 0x7f041c09c870 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f0421c63640 10 monclient(hunting): authenticate will time out at 772.870622s 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f041bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f041c0b4360 0x7f041c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60302/0 (socket says 172.21.15.136:60302) 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.550+0000 7f041bfff640 1 -- 172.21.15.136:0/2274576953 learned_addr learned my addr 172.21.15.136:0/2274576953 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573481614 0 0) 0x7f041c09da30 con 0x7f041c0b4360 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0408003520 con 0x7f041c0b4360 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662023756 0 0) 0x7f041c0b39c0 con 0x7f041c09c870 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f041c09da30 con 0x7f041c09c870 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801468957 0 0) 0x7f040c001040 con 0x7f041c09ac40 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f041c0b39c0 con 0x7f041c09ac40 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2385959145 0 0) 0x7f0408003520 con 0x7f041c0b4360 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 >> v1:172.21.15.117:6790/0 conn(0x7f041c09c870 legacy=0x7f041c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 >> v1:172.21.15.117:6789/0 conn(0x7f041c09ac40 legacy=0x7f041c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 monclient: found mon.noname-c 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:15.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c0b5160 con 0x7f041c0b4360 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: _finish_auth 0 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0414001a10 con 0x7f041c0b4360 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f0421c63640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f0421c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: dump: 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.551 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.551 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.551 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.551 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.551 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.551 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.551 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.551 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f0421c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0414001bf0 con 0x7f041c0b4360 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f0421c63640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f0421c63640 4 set_mon_vals no callback set 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 1 -- 172.21.15.136:0/2274576953 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0414002000 con 0x7f041c0b4360 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.551+0000 7f041affd640 10 monclient: dump: 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.552 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.553 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.552+0000 7f0421c63640 10 monclient: shutdown 2024-03-29T10:34:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.552+0000 7f0421c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.552+0000 7f0421c63640 1 -- 172.21.15.136:0/2274576953 >> v1:172.21.15.136:6789/0 conn(0x7f041c0b4360 legacy=0x7f041c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.552+0000 7f0421c63640 1 -- 172.21.15.136:0/2274576953 shutdown_connections 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.552+0000 7f0421c63640 1 -- 172.21.15.136:0/2274576953 >> 172.21.15.136:0/2274576953 conn(0x7f041c066b00 msgr2=0x7f041c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.552+0000 7f0421c63640 1 -- 172.21.15.136:0/2274576953 shutdown_connections 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.552+0000 7f0421c63640 1 -- 172.21.15.136:0/2274576953 wait complete. 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.552+0000 7f0421c63640 10 monclient: build_initial_monmap 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.552+0000 7f0421c63640 10 monclient: monmap: 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 1 librados: starting msgr at 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 1 librados: starting objecter 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 1 Processor -- start 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 1 -- start start 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 1 librados: setting wanted keys 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 1 librados: calling monclient init 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 10 monclient: init 2024-03-29T10:34:15.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 10 monclient(hunting): picked mon.noname-c con 0x7f041c09ac40 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 10 monclient(hunting): picked mon.noname-a con 0x7f041c0b04a0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 10 monclient(hunting): picked mon.noname-b con 0x7f041c1d7ce0 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041c140490 con 0x7f041c0b04a0 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041c0b0980 con 0x7f041c1d7ce0 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041c0b1a80 con 0x7f041c09ac40 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0421c63640 10 monclient(hunting): authenticate will time out at 772.873617s 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f041bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f041c1d7ce0 0x7f041c1e2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60314/0 (socket says 172.21.15.136:60314) 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f041bfff640 1 -- 172.21.15.136:0/3672435442 learned_addr learned my addr 172.21.15.136:0/3672435442 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0418ff9640 10 client.?.objecter ms_handle_connect 0x7f041c1d7ce0 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0418ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:15.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0418ff9640 10 client.?.objecter ms_handle_connect 0x7f041c0b04a0 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0418ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0418ff9640 10 client.?.objecter ms_handle_connect 0x7f041c09ac40 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.553+0000 7f0418ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770840463 0 0) 0x7f041c0b0980 con 0x7f041c1d7ce0 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03fc004eb0 con 0x7f041c1d7ce0 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549755112 0 0) 0x7f041c140490 con 0x7f041c0b04a0 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f041c0b0980 con 0x7f041c0b04a0 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486023060 0 0) 0x7f041c0b1a80 con 0x7f041c09ac40 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f041c140490 con 0x7f041c09ac40 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3171207089 0 0) 0x7f03fc004eb0 con 0x7f041c1d7ce0 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f041c0b1a80 con 0x7f041c1d7ce0 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0414001f40 con 0x7f041c1d7ce0 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:15.558 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2483689050 0 0) 0x7f041c0b0980 con 0x7f041c0b04a0 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03fc004eb0 con 0x7f041c0b04a0 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f04040018f0 con 0x7f041c0b04a0 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.561 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773356809 0 0) 0x7f041c140490 con 0x7f041c09ac40 2024-03-29T10:34:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f041c0b0980 con 0x7f041c09ac40 2024-03-29T10:34:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f040c001650 con 0x7f041c09ac40 2024-03-29T10:34:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:15.562 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.562 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.562 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.562 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.562 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.563 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.563 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.563 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.563 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.563 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3871156755 0 0) 0x7f03fc004eb0 con 0x7f041c0b04a0 2024-03-29T10:34:15.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 >> v1:172.21.15.117:6790/0 conn(0x7f041c09ac40 legacy=0x7f041c0afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 >> v1:172.21.15.136:6789/0 conn(0x7f041c1d7ce0 legacy=0x7f041c1e2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:34:15.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 monclient: found mon.a 2024-03-29T10:34:15.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:15.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:15.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c0b0ea0 con 0x7f041c0b04a0 2024-03-29T10:34:15.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient: _finish_auth 0 2024-03-29T10:34:15.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:34:15.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:15.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0421c63640 5 monclient: authenticate success, global_id 4119 2024-03-29T10:34:15.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0421c63640 10 monclient: _renew_subs 2024-03-29T10:34:15.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0421c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:15.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0421c63640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041c1e3730 con 0x7f041c0b04a0 2024-03-29T10:34:15.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0421c63640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:15.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0421c63640 10 monclient: _renew_subs 2024-03-29T10:34:15.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0421c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:15.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0421c63640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f041c1e3aa0 con 0x7f041c0b04a0 2024-03-29T10:34:15.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0421c63640 1 librados: init done 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0404001d60 con 0x7f041c0b04a0 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0404003790 con 0x7f041c0b04a0 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f041a7fc640 4 set_mon_vals no callback set 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 monclient: dump: 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0404003c80 con 0x7f041c0b04a0 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 439952768 0 0) 0x7f0404005300 con 0x7f041c0b04a0 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 10 client.4119.objecter ms_dispatch 0x7f041c000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.554+0000 7f0418ff9640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.555+0000 7f0421c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.555+0000 7f0421c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.555+0000 7f0421c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.555+0000 7f0421c63640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0400005080 con 0x7f041c0b04a0 2024-03-29T10:34:15.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.557+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0404004160 con 0x7f041c0b04a0 2024-03-29T10:34:15.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.557+0000 7f0418ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:15.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.557+0000 7f0418ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:15.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.629+0000 7f0421c63640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "af857458-18ba-48bb-ba1c-5577c5a7ff12", "id": 0}] 2024-03-29T10:34:15.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.629+0000 7f0421c63640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "af857458-18ba-48bb-ba1c-5577c5a7ff12", "id": 0}] 2024-03-29T10:34:15.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.629+0000 7f0421c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:15.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.629+0000 7f0421c63640 1 -- 172.21.15.136:0/3672435442 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd new", "uuid": "af857458-18ba-48bb-ba1c-5577c5a7ff12", "id": 0} v 0) v1 -- 0x7f04000058b0 con 0x7f041c0b04a0 2024-03-29T10:34:15.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.630+0000 7f0418ff9640 1 -- 172.21.15.136:0/3672435442 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "af857458-18ba-48bb-ba1c-5577c5a7ff12", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 773824465 0 732707515) 0x7f04040043d0 con 0x7f041c0b04a0 2024-03-29T10:34:15.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.630+0000 7f0418ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "af857458-18ba-48bb-ba1c-5577c5a7ff12", "id": 0}] 2024-03-29T10:34:15.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.630+0000 7f0418ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:15.616 INFO:teuthology.orchestra.run.smithi136.stdout:0 2024-03-29T10:34:15.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 10 librados: watch_flush enter 2024-03-29T10:34:15.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 10 librados: watch_flush exit 2024-03-29T10:34:15.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-29T10:34:15.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 10 client.4119.objecter successfully canceled tick 2024-03-29T10:34:15.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 10 monclient: shutdown 2024-03-29T10:34:15.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:15.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 1 -- 172.21.15.136:0/3672435442 >> v1:172.21.15.117:6789/0 conn(0x7f041c0b04a0 legacy=0x7f041c1d75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 1 -- 172.21.15.136:0/3672435442 shutdown_connections 2024-03-29T10:34:15.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 1 -- 172.21.15.136:0/3672435442 >> 172.21.15.136:0/3672435442 conn(0x7f041c066b00 msgr2=0x7f041c13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:15.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 1 -- 172.21.15.136:0/3672435442 shutdown_connections 2024-03-29T10:34:15.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 1 -- 172.21.15.136:0/3672435442 wait complete. 2024-03-29T10:34:15.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.631+0000 7f0421c63640 1 librados: shutdown 2024-03-29T10:34:15.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 265a899d-aeb3-4ec8-a4df-5ebd03678897 1 2024-03-29T10:34:15.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.742+0000 7fab4e5b6640 10 monclient: get_monmap_and_config 2024-03-29T10:34:15.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.742+0000 7fab4e5b6640 10 monclient: build_initial_monmap 2024-03-29T10:34:15.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.742+0000 7fab4e5b6640 10 monclient: monmap: 2024-03-29T10:34:15.727 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:15.727 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.727 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:34:15.742950+0000 2024-03-29T10:34:15.727 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:34:15.742950+0000 2024-03-29T10:34:15.728 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:15.728 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.728 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:15.728 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:15.728 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:15.728 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.742+0000 7fab4e5b6640 1 Processor -- start 2024-03-29T10:34:15.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.742+0000 7fab4e5b6640 1 -- start start 2024-03-29T10:34:15.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.742+0000 7fab4e5b6640 10 monclient: init 2024-03-29T10:34:15.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:15.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:34:15.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 10 monclient(hunting): picked mon.noname-a con 0x7fab4809ac40 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 10 monclient(hunting): picked mon.noname-b con 0x7fab4809c870 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 10 monclient(hunting): picked mon.noname-c con 0x7fab480b4360 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4809dec0 con 0x7fab4809ac40 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4809da30 con 0x7fab480b4360 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480b39c0 con 0x7fab4809c870 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4e5b6640 10 monclient(hunting): authenticate will time out at 773.063447s 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4cb53640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fab480b4360 0x7fab4814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60320/0 (socket says 172.21.15.136:60320) 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab4cb53640 1 -- 172.21.15.136:0/1484271290 learned_addr learned my addr 172.21.15.136:0/1484271290 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471186404 0 0) 0x7fab4809da30 con 0x7fab480b4360 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab46ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab46ffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.743+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab30003520 con 0x7fab480b4360 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051169898 0 0) 0x7fab3c001040 con 0x7fab4809ac40 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4809da30 con 0x7fab4809ac40 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596629634 0 0) 0x7fab480b39c0 con 0x7fab4809c870 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:34:15.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab3c001040 con 0x7fab4809c870 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2102628292 0 0) 0x7fab30003520 con 0x7fab480b4360 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 >> v1:172.21.15.117:6790/0 conn(0x7fab4809c870 legacy=0x7fab4813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 >> v1:172.21.15.117:6789/0 conn(0x7fab4809ac40 legacy=0x7fab480b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 monclient: found mon.noname-c 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab480b5160 con 0x7fab480b4360 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: _finish_auth 0 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fab40001a10 con 0x7fab480b4360 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab4e5b6640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab4e5b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: dump: 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.741 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab4e5b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab40001bf0 con 0x7fab480b4360 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 1 -- 172.21.15.136:0/1484271290 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fab40002000 con 0x7fab480b4360 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab46ffd640 10 monclient: dump: 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.742 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab4e5b6640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.744+0000 7fab4e5b6640 4 set_mon_vals no callback set 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.745+0000 7fab4e5b6640 10 monclient: shutdown 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.745+0000 7fab4e5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.745+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1484271290 >> v1:172.21.15.136:6789/0 conn(0x7fab480b4360 legacy=0x7fab4814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.745+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1484271290 shutdown_connections 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.745+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1484271290 >> 172.21.15.136:0/1484271290 conn(0x7fab48066b00 msgr2=0x7fab48066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.745+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1484271290 shutdown_connections 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.745+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1484271290 wait complete. 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.746+0000 7fab4e5b6640 10 monclient: build_initial_monmap 2024-03-29T10:34:15.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.746+0000 7fab4e5b6640 10 monclient: monmap: 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.746+0000 7fab4e5b6640 1 librados: starting msgr at 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.746+0000 7fab4e5b6640 1 librados: starting objecter 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.746+0000 7fab4e5b6640 1 Processor -- start 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.746+0000 7fab4e5b6640 1 -- start start 2024-03-29T10:34:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.746+0000 7fab4e5b6640 1 librados: setting wanted keys 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.746+0000 7fab4e5b6640 1 librados: calling monclient init 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.746+0000 7fab4e5b6640 10 monclient: init 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 10 monclient(hunting): picked mon.noname-a con 0x7fab4809ac40 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 10 monclient(hunting): picked mon.noname-b con 0x7fab48141910 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 10 monclient(hunting): picked mon.noname-c con 0x7fab4813ffa0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab48139b60 con 0x7fab4809ac40 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab481d6f10 con 0x7fab48141910 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480afba0 con 0x7fab4813ffa0 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab4e5b6640 10 monclient(hunting): authenticate will time out at 773.067793s 2024-03-29T10:34:15.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab477fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fab48141910 0x7fab481d73a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60324/0 (socket says 172.21.15.136:60324) 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.747+0000 7fab477fe640 1 -- 172.21.15.136:0/1614882174 learned_addr learned my addr 172.21.15.136:0/1614882174 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 client.?.objecter ms_handle_connect 0x7fab48141910 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 client.?.objecter ms_handle_connect 0x7fab4813ffa0 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 client.?.objecter ms_handle_connect 0x7fab4809ac40 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333835008 0 0) 0x7fab481d6f10 con 0x7fab48141910 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:34:15.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34004eb0 con 0x7fab48141910 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343669470 0 0) 0x7fab480afba0 con 0x7fab4813ffa0 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab481d6f10 con 0x7fab4813ffa0 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895491210 0 0) 0x7fab4809da30 con 0x7fab4809ac40 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab480afba0 con 0x7fab4809ac40 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2540295446 0 0) 0x7fab481d6f10 con 0x7fab4813ffa0 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab4809da30 con 0x7fab4813ffa0 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fab40001f00 con 0x7fab4813ffa0 2024-03-29T10:34:15.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2697186335 0 0) 0x7fab480afba0 con 0x7fab4809ac40 2024-03-29T10:34:15.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab481d6f10 con 0x7fab4809ac40 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 970674133 0 0) 0x7fab34004eb0 con 0x7fab48141910 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab480afba0 con 0x7fab48141910 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fab3c0017d0 con 0x7fab4809ac40 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.749 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fab38001990 con 0x7fab48141910 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.750 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4239160273 0 0) 0x7fab4809da30 con 0x7fab4813ffa0 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 >> v1:172.21.15.136:6789/0 conn(0x7fab48141910 legacy=0x7fab481d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 >> v1:172.21.15.117:6789/0 conn(0x7fab4809ac40 legacy=0x7fab48141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 monclient: found mon.c 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:15.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab48139520 con 0x7fab4813ffa0 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient: _finish_auth 0 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab4e5b6640 5 monclient: authenticate success, global_id 4127 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab4e5b6640 10 monclient: _renew_subs 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab4e5b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab48139710 con 0x7fab4813ffa0 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab4e5b6640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab4e5b6640 10 monclient: _renew_subs 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab4e5b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab481d7c90 con 0x7fab4813ffa0 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab4e5b6640 1 librados: init done 2024-03-29T10:34:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab40004e10 con 0x7fab4813ffa0 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fab40005120 con 0x7fab4813ffa0 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab467fc640 4 set_mon_vals no callback set 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 monclient: dump: 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.753 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.754 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fab40005610 con 0x7fab4813ffa0 2024-03-29T10:34:15.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3653041562 0 0) 0x7fab40006a60 con 0x7fab4813ffa0 2024-03-29T10:34:15.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 10 client.4127.objecter ms_dispatch 0x7fab48000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:34:15.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 3 client.4127.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:34:15.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 3 client.4127.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:34:15.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.748+0000 7fab44ff9640 20 client.4127.objecter dump_active .. 0 homeless 2024-03-29T10:34:15.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.749+0000 7fab4e5b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:15.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.749+0000 7fab4e5b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:15.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.749+0000 7fab4e5b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:15.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.749+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab28005080 con 0x7fab4813ffa0 2024-03-29T10:34:15.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.751+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fab40005a60 con 0x7fab4813ffa0 2024-03-29T10:34:15.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.751+0000 7fab44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:15.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.751+0000 7fab44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:15.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.824+0000 7fab4e5b6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "265a899d-aeb3-4ec8-a4df-5ebd03678897", "id": 1}] 2024-03-29T10:34:15.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.824+0000 7fab4e5b6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "265a899d-aeb3-4ec8-a4df-5ebd03678897", "id": 1}] 2024-03-29T10:34:15.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.824+0000 7fab4e5b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:15.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.824+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1614882174 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd new", "uuid": "265a899d-aeb3-4ec8-a4df-5ebd03678897", "id": 1} v 0) v1 -- 0x7fab280058b0 con 0x7fab4813ffa0 2024-03-29T10:34:15.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.826+0000 7fab44ff9640 1 -- 172.21.15.136:0/1614882174 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "265a899d-aeb3-4ec8-a4df-5ebd03678897", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3845442898 0 940483276) 0x7fab40005cd0 con 0x7fab4813ffa0 2024-03-29T10:34:15.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.826+0000 7fab44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "265a899d-aeb3-4ec8-a4df-5ebd03678897", "id": 1}] 2024-03-29T10:34:15.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.826+0000 7fab44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:15.812 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-03-29T10:34:15.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 10 librados: watch_flush enter 2024-03-29T10:34:15.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 10 librados: watch_flush exit 2024-03-29T10:34:15.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 20 client.4127.objecter shutdown clearing up homeless session... 2024-03-29T10:34:15.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 10 client.4127.objecter successfully canceled tick 2024-03-29T10:34:15.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 10 monclient: shutdown 2024-03-29T10:34:15.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:15.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1614882174 >> v1:172.21.15.117:6790/0 conn(0x7fab4813ffa0 legacy=0x7fab48138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1614882174 shutdown_connections 2024-03-29T10:34:15.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1614882174 >> 172.21.15.136:0/1614882174 conn(0x7fab48066b00 msgr2=0x7fab4813c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:15.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.827+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1614882174 shutdown_connections 2024-03-29T10:34:15.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.828+0000 7fab4e5b6640 1 -- 172.21.15.136:0/1614882174 wait complete. 2024-03-29T10:34:15.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.828+0000 7fab4e5b6640 1 librados: shutdown 2024-03-29T10:34:15.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new ccdfb173-0663-4a89-b597-18562ec7c9c9 2 2024-03-29T10:34:15.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.939+0000 7f639132b640 10 monclient: get_monmap_and_config 2024-03-29T10:34:15.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.939+0000 7f639132b640 10 monclient: build_initial_monmap 2024-03-29T10:34:15.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.939+0000 7f639132b640 10 monclient: monmap: 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:34:15.939907+0000 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:34:15.939907+0000 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.939+0000 7f639132b640 1 Processor -- start 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.939+0000 7f639132b640 1 -- start start 2024-03-29T10:34:15.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.939+0000 7f639132b640 10 monclient: init 2024-03-29T10:34:15.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:15.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:15.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 10 monclient(hunting): picked mon.noname-a con 0x7f638c0af360 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:15.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 10 monclient(hunting): picked mon.noname-c con 0x7f638c06af60 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 10 monclient(hunting): picked mon.noname-b con 0x7f638c145e60 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0b0520 con 0x7f638c0af360 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0aff20 con 0x7f638c06af60 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0b1da0 con 0x7f638c145e60 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 10 monclient(hunting): authenticate will time out at 773.260197s 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f638a7fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f638c06af60 0x7f638c145750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60332/0 (socket says 172.21.15.136:60332) 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f638a7fc640 1 -- 172.21.15.136:0/704758176 learned_addr learned my addr 172.21.15.136:0/704758176 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881515383 0 0) 0x7f638c0aff20 con 0x7f638c06af60 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:34:15.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6374003520 con 0x7f638c06af60 2024-03-29T10:34:15.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592083393 0 0) 0x7f638c0b1da0 con 0x7f638c145e60 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c0aff20 con 0x7f638c145e60 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918302191 0 0) 0x7f638c0b0520 con 0x7f638c0af360 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c0b1da0 con 0x7f638c0af360 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3469576160 0 0) 0x7f6374003520 con 0x7f638c06af60 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 >> v1:172.21.15.117:6790/0 conn(0x7f638c145e60 legacy=0x7f638c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 >> v1:172.21.15.117:6789/0 conn(0x7f638c0af360 legacy=0x7f638c06c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 monclient: found mon.noname-c 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c0b2080 con 0x7f638c06af60 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: _finish_auth 0 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6380001690 con 0x7f638c06af60 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: dump: 2024-03-29T10:34:15.939 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6380001870 con 0x7f638c06af60 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 1 -- 172.21.15.136:0/704758176 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6380001c00 con 0x7f638c06af60 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f639132b640 4 set_mon_vals no callback set 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.940+0000 7f6389ffb640 10 monclient: dump: 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.941+0000 7f639132b640 10 monclient: shutdown 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.941+0000 7f639132b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.941+0000 7f639132b640 1 -- 172.21.15.136:0/704758176 >> v1:172.21.15.136:6789/0 conn(0x7f638c06af60 legacy=0x7f638c145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.941+0000 7f639132b640 1 -- 172.21.15.136:0/704758176 shutdown_connections 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.941+0000 7f639132b640 1 -- 172.21.15.136:0/704758176 >> 172.21.15.136:0/704758176 conn(0x7f638c0b58f0 msgr2=0x7f638c0b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.941+0000 7f639132b640 1 -- 172.21.15.136:0/704758176 shutdown_connections 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.941+0000 7f639132b640 1 -- 172.21.15.136:0/704758176 wait complete. 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.942+0000 7f639132b640 10 monclient: build_initial_monmap 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.942+0000 7f639132b640 10 monclient: monmap: 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T10:34:15.942 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.942+0000 7f639132b640 1 librados: starting msgr at 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.942+0000 7f639132b640 1 librados: starting objecter 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.942+0000 7f639132b640 1 Processor -- start 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.942+0000 7f639132b640 1 -- start start 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.942+0000 7f639132b640 1 librados: setting wanted keys 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.942+0000 7f639132b640 1 librados: calling monclient init 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.942+0000 7f639132b640 10 monclient: init 2024-03-29T10:34:15.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 10 monclient(hunting): picked mon.noname-a con 0x7f638c06af60 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 10 monclient(hunting): picked mon.noname-c con 0x7f638c0af360 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 10 monclient(hunting): picked mon.noname-b con 0x7f638c145e60 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c123ed0 con 0x7f638c06af60 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c1d6e30 con 0x7f638c145e60 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c11ee90 con 0x7f638c0af360 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f639132b640 10 monclient(hunting): authenticate will time out at 773.263277s 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f638b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f638c145e60 0x7f638c1d7310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60348/0 (socket says 172.21.15.136:60348) 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f638a7fc640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f638c0af360 0x7f638c122850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.136:48482/0 (socket says 172.21.15.136:48482) 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f638b7fe640 1 -- 172.21.15.136:0/145147186 learned_addr learned my addr 172.21.15.136:0/145147186 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 client.?.objecter ms_handle_connect 0x7f638c145e60 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 client.?.objecter ms_handle_connect 0x7f638c0af360 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 client.?.objecter ms_handle_connect 0x7f638c06af60 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041471699 0 0) 0x7f638c11ee90 con 0x7f638c0af360 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6378004eb0 con 0x7f638c0af360 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787508846 0 0) 0x7f638c1d6e30 con 0x7f638c145e60 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c11ee90 con 0x7f638c145e60 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610761208 0 0) 0x7f638c123ed0 con 0x7f638c06af60 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c1d6e30 con 0x7f638c06af60 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3395852055 0 0) 0x7f6378004eb0 con 0x7f638c0af360 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f638c123ed0 con 0x7f638c0af360 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6380001b80 con 0x7f638c0af360 2024-03-29T10:34:15.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): dump: 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:15.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3203515314 0 0) 0x7f638c11ee90 con 0x7f638c145e60 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6378004eb0 con 0x7f638c145e60 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6384001b80 con 0x7f638c145e60 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 10 monclient(hunting): dump: 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.948 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.943+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3333309609 0 0) 0x7f638c1d6e30 con 0x7f638c06af60 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f638c11ee90 con 0x7f638c06af60 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f637c001830 con 0x7f638c06af60 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient(hunting): dump: 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.949 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4277129194 0 0) 0x7f638c123ed0 con 0x7f638c0af360 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 >> v1:172.21.15.136:6789/0 conn(0x7f638c145e60 legacy=0x7f638c1d7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 >> v1:172.21.15.117:6789/0 conn(0x7f638c06af60 legacy=0x7f638c121ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient: _finish_hunting 0 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 monclient: found mon.c 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c11f020 con 0x7f638c0af360 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient: _finish_auth 0 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient: _check_auth_tickets 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f639132b640 5 monclient: authenticate success, global_id 4133 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f639132b640 10 monclient: _renew_subs 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f639132b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f639132b640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638c1d7a20 con 0x7f638c0af360 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f639132b640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f639132b640 10 monclient: _renew_subs 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f639132b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f639132b640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f638c140bc0 con 0x7f638c0af360 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f639132b640 1 librados: init done 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6380004a90 con 0x7f638c0af360 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6380004da0 con 0x7f638c0af360 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:15.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 monclient: dump: 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6380005290 con 0x7f638c0af360 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63897fa640 4 set_mon_vals no callback set 2024-03-29T10:34:15.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2295005131 0 0) 0x7f6380005960 con 0x7f638c0af360 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 10 client.4133.objecter ms_dispatch 0x7f638c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 3 client.4133.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 3 client.4133.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.944+0000 7f63737fe640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.945+0000 7f639132b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.945+0000 7f639132b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.945+0000 7f639132b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.945+0000 7f639132b640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638c067c00 con 0x7f638c0af360 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.947+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6380005c40 con 0x7f638c0af360 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.947+0000 7f63737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:15.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:15.947+0000 7f63737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:15.986 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.000+0000 7f414a1ed040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:34:15.998 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.013+0000 7f2e34034040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:34:16.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.019+0000 7f639132b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ccdfb173-0663-4a89-b597-18562ec7c9c9", "id": 2}] 2024-03-29T10:34:16.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.019+0000 7f639132b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ccdfb173-0663-4a89-b597-18562ec7c9c9", "id": 2}] 2024-03-29T10:34:16.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.019+0000 7f639132b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.019+0000 7f639132b640 1 -- 172.21.15.136:0/145147186 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd new", "uuid": "ccdfb173-0663-4a89-b597-18562ec7c9c9", "id": 2} v 0) v1 -- 0x7f638c04f730 con 0x7f638c0af360 2024-03-29T10:34:16.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.023+0000 7f63737fe640 1 -- 172.21.15.136:0/145147186 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ccdfb173-0663-4a89-b597-18562ec7c9c9", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1408915653 0 216599125) 0x7f6380005540 con 0x7f638c0af360 2024-03-29T10:34:16.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.023+0000 7f63737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ccdfb173-0663-4a89-b597-18562ec7c9c9", "id": 2}] 2024-03-29T10:34:16.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.023+0000 7f63737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:16.008 INFO:teuthology.orchestra.run.smithi136.stdout:2 2024-03-29T10:34:16.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.023+0000 7f639132b640 10 librados: watch_flush enter 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.023+0000 7f639132b640 10 librados: watch_flush exit 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.023+0000 7f639132b640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.023+0000 7f639132b640 10 client.4133.objecter successfully canceled tick 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.024+0000 7f639132b640 10 monclient: shutdown 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.024+0000 7f639132b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.024+0000 7f639132b640 1 -- 172.21.15.136:0/145147186 >> v1:172.21.15.117:6790/0 conn(0x7f638c0af360 legacy=0x7f638c122850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.024+0000 7f639132b640 1 -- 172.21.15.136:0/145147186 shutdown_connections 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.024+0000 7f639132b640 1 -- 172.21.15.136:0/145147186 >> 172.21.15.136:0/145147186 conn(0x7f638c0b58f0 msgr2=0x7f638c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.024+0000 7f639132b640 1 -- 172.21.15.136:0/145147186 shutdown_connections 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.024+0000 7f639132b640 1 -- 172.21.15.136:0/145147186 wait complete. 2024-03-29T10:34:16.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.024+0000 7f639132b640 1 librados: shutdown 2024-03-29T10:34:16.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 89e22417-65aa-443b-8c73-89217c69baaf 3 2024-03-29T10:34:16.040 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.055+0000 7f414a1ed040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:34:16.056 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.071+0000 7f2e34034040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:34:16.084 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.099+0000 7f414a1ed040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:34:16.103 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.118+0000 7f2e34034040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:34:16.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.134+0000 7f5c4250e640 10 monclient: get_monmap_and_config 2024-03-29T10:34:16.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.134+0000 7f5c4250e640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.134+0000 7f5c4250e640 10 monclient: monmap: 2024-03-29T10:34:16.119 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.119 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.119 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:34:16.135087+0000 2024-03-29T10:34:16.120 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:34:16.135087+0000 2024-03-29T10:34:16.120 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.120 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.120 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.120 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:16.120 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:16.120 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.134+0000 7f5c4250e640 1 Processor -- start 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.134+0000 7f5c4250e640 1 -- start start 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.134+0000 7f5c4250e640 10 monclient: init 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 10 monclient(hunting): picked mon.noname-c con 0x7f5c3c09ac40 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 10 monclient(hunting): picked mon.noname-b con 0x7f5c3c09c870 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 10 monclient(hunting): picked mon.noname-a con 0x7f5c3c0b4360 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c09dec0 con 0x7f5c3c0b4360 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c09da30 con 0x7f5c3c09ac40 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c0b39c0 con 0x7f5c3c09c870 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c4250e640 10 monclient(hunting): authenticate will time out at 773.455494s 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c3bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5c3c09ac40 0x7f5c3c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60356/0 (socket says 172.21.15.136:60356) 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c3bfff640 1 -- 172.21.15.136:0/1223704782 learned_addr learned my addr 172.21.15.136:0/1223704782 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908389934 0 0) 0x7f5c3c09da30 con 0x7f5c3c09ac40 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c3affd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c3affd640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.135+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c24003520 con 0x7f5c3c09ac40 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455191381 0 0) 0x7f5c340013c0 con 0x7f5c3c0b4360 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c3c09da30 con 0x7f5c3c0b4360 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082774566 0 0) 0x7f5c3c0b39c0 con 0x7f5c3c09c870 2024-03-29T10:34:16.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c340013c0 con 0x7f5c3c09c870 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 862119578 0 0) 0x7f5c24003520 con 0x7f5c3c09ac40 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 >> v1:172.21.15.117:6790/0 conn(0x7f5c3c09c870 legacy=0x7f5c3c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 >> v1:172.21.15.117:6789/0 conn(0x7f5c3c0b4360 legacy=0x7f5c3c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 monclient: found mon.noname-c 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c0b5160 con 0x7f5c3c09ac40 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5c30001690 con 0x7f5c3c09ac40 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: dump: 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.132 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c30001870 con 0x7f5c3c09ac40 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 1 -- 172.21.15.136:0/1223704782 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5c30001c80 con 0x7f5c3c09ac40 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 4 set_mon_vals no callback set 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c3affd640 10 monclient: dump: 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.133 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 10 monclient: shutdown 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 1 -- 172.21.15.136:0/1223704782 >> v1:172.21.15.136:6789/0 conn(0x7f5c3c09ac40 legacy=0x7f5c3c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 1 -- 172.21.15.136:0/1223704782 shutdown_connections 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.136+0000 7f5c4250e640 1 -- 172.21.15.136:0/1223704782 >> 172.21.15.136:0/1223704782 conn(0x7f5c3c066b00 msgr2=0x7f5c3c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.137+0000 7f5c4250e640 1 -- 172.21.15.136:0/1223704782 shutdown_connections 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.137+0000 7f5c4250e640 1 -- 172.21.15.136:0/1223704782 wait complete. 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.137+0000 7f5c4250e640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.137+0000 7f5c4250e640 10 monclient: monmap: 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.137+0000 7f5c4250e640 1 librados: starting msgr at 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.137+0000 7f5c4250e640 1 librados: starting objecter 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.137+0000 7f5c4250e640 1 Processor -- start 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 1 -- start start 2024-03-29T10:34:16.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 1 librados: setting wanted keys 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 1 librados: calling monclient init 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 10 monclient: init 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 10 monclient(hunting): picked mon.noname-b con 0x7f5c3c09ac40 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 10 monclient(hunting): picked mon.noname-a con 0x7f5c3c0b1860 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 10 monclient(hunting): picked mon.noname-c con 0x7f5c3c13b5b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c1d7660 con 0x7f5c3c0b1860 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c0ade20 con 0x7f5c3c09ac40 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c1406b0 con 0x7f5c3c13b5b0 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c4250e640 10 monclient(hunting): authenticate will time out at 773.458660s 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c3bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5c3c09ac40 0x7f5c3c13ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60358/0 (socket says 172.21.15.136:60358) 2024-03-29T10:34:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c3bfff640 1 -- 172.21.15.136:0/637128958 learned_addr learned my addr 172.21.15.136:0/637128958 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c38ff9640 10 client.?.objecter ms_handle_connect 0x7f5c3c09ac40 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c38ff9640 10 client.?.objecter ms_handle_connect 0x7f5c3c0b1860 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c38ff9640 10 client.?.objecter ms_handle_connect 0x7f5c3c13b5b0 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.138+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618293777 0 0) 0x7f5c3c0ade20 con 0x7f5c3c09ac40 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c28004eb0 con 0x7f5c3c09ac40 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201619817 0 0) 0x7f5c3c1406b0 con 0x7f5c3c13b5b0 2024-03-29T10:34:16.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c3c0ade20 con 0x7f5c3c13b5b0 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193333687 0 0) 0x7f5c3c1d7660 con 0x7f5c3c0b1860 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c3c1406b0 con 0x7f5c3c0b1860 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1375380073 0 0) 0x7f5c28004eb0 con 0x7f5c3c09ac40 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c3c1d7660 con 0x7f5c3c09ac40 2024-03-29T10:34:16.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5c30001bc0 con 0x7f5c3c09ac40 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:16.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1595239092 0 0) 0x7f5c3c0ade20 con 0x7f5c3c13b5b0 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c28004eb0 con 0x7f5c3c13b5b0 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4237861499 0 0) 0x7f5c3c1406b0 con 0x7f5c3c0b1860 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c3c0ade20 con 0x7f5c3c0b1860 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5c34001ab0 con 0x7f5c3c13b5b0 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.140 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5c2c0017f0 con 0x7f5c3c0b1860 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.141 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 547862435 0 0) 0x7f5c3c1d7660 con 0x7f5c3c09ac40 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 >> v1:172.21.15.117:6790/0 conn(0x7f5c3c13b5b0 legacy=0x7f5c3c1e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 >> v1:172.21.15.117:6789/0 conn(0x7f5c3c0b1860 legacy=0x7f5c3c13aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 monclient: found mon.b 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c1cccd0 con 0x7f5c3c09ac40 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c4250e640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:34:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c4250e640 10 monclient: _renew_subs 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c4250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c4250e640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3c1cce60 con 0x7f5c3c09ac40 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c4250e640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c4250e640 10 monclient: _renew_subs 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c4250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c4250e640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c1d7190 con 0x7f5c3c09ac40 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c4250e640 1 librados: init done 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c30004bf0 con 0x7f5c3c09ac40 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5c30004f00 con 0x7f5c3c09ac40 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c3a7fc640 4 set_mon_vals no callback set 2024-03-29T10:34:16.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 monclient: dump: 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5c300053f0 con 0x7f5c3c09ac40 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2051106348 0 0) 0x7f5c30006970 con 0x7f5c3c09ac40 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 10 client.4132.objecter ms_dispatch 0x7f5c3c000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:34:16.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:34:16.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.139+0000 7f5c38ff9640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T10:34:16.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.140+0000 7f5c4250e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.140+0000 7f5c4250e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.140+0000 7f5c4250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.140+0000 7f5c4250e640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1c005080 con 0x7f5c3c09ac40 2024-03-29T10:34:16.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.142+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5c300064e0 con 0x7f5c3c09ac40 2024-03-29T10:34:16.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.142+0000 7f5c38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.142+0000 7f5c38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:16.145 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.153+0000 7f414a1ed040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:34:16.160 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.175+0000 7f2e34034040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:34:16.180 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.195+0000 7f414a1ed040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:34:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.215+0000 7f5c4250e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "89e22417-65aa-443b-8c73-89217c69baaf", "id": 3}] 2024-03-29T10:34:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.215+0000 7f5c4250e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "89e22417-65aa-443b-8c73-89217c69baaf", "id": 3}] 2024-03-29T10:34:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.215+0000 7f5c4250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.215+0000 7f5c4250e640 1 -- 172.21.15.136:0/637128958 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "89e22417-65aa-443b-8c73-89217c69baaf", "id": 3} v 0) v1 -- 0x7f5c1c0058b0 con 0x7f5c3c09ac40 2024-03-29T10:34:16.204 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.219+0000 7f2e34034040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:34:16.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.222+0000 7f5c38ff9640 1 -- 172.21.15.136:0/637128958 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89e22417-65aa-443b-8c73-89217c69baaf", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3373615344 0 525046306) 0x7f5c30005790 con 0x7f5c3c09ac40 2024-03-29T10:34:16.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.222+0000 7f5c38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "89e22417-65aa-443b-8c73-89217c69baaf", "id": 3}] 2024-03-29T10:34:16.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.222+0000 7f5c38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:16.208 INFO:teuthology.orchestra.run.smithi136.stdout:3 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 10 librados: watch_flush enter 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 10 librados: watch_flush exit 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 10 client.4132.objecter successfully canceled tick 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 10 monclient: shutdown 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 1 -- 172.21.15.136:0/637128958 >> v1:172.21.15.136:6789/0 conn(0x7f5c3c09ac40 legacy=0x7f5c3c13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 1 -- 172.21.15.136:0/637128958 shutdown_connections 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 1 -- 172.21.15.136:0/637128958 >> 172.21.15.136:0/637128958 conn(0x7f5c3c066b00 msgr2=0x7f5c3c13cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 1 -- 172.21.15.136:0/637128958 shutdown_connections 2024-03-29T10:34:16.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 1 -- 172.21.15.136:0/637128958 wait complete. 2024-03-29T10:34:16.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.223+0000 7f5c4250e640 1 librados: shutdown 2024-03-29T10:34:16.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 31b056b7-6332-4aa0-a4da-21f1f5c7f410 4 2024-03-29T10:34:16.295 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.310+0000 7f414a1ed040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:34:16.327 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.342+0000 7f2e34034040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:34:16.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.375+0000 7f6e1e814640 10 monclient: get_monmap_and_config 2024-03-29T10:34:16.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.375+0000 7f6e1e814640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.375+0000 7f6e1e814640 10 monclient: monmap: 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:34:16.376166+0000 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:34:16.376166+0000 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:16.361 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.368 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.379+0000 7f414a1ed040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:34:16.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 1 Processor -- start 2024-03-29T10:34:16.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 1 -- start start 2024-03-29T10:34:16.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 10 monclient: init 2024-03-29T10:34:16.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 10 monclient(hunting): picked mon.noname-c con 0x7f6e1809ac40 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 10 monclient(hunting): picked mon.noname-a con 0x7f6e1809c870 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 10 monclient(hunting): picked mon.noname-b con 0x7f6e180b4360 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1809dec0 con 0x7f6e1809c870 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1809da30 con 0x7f6e1809ac40 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e180b39c0 con 0x7f6e180b4360 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e1e814640 10 monclient(hunting): authenticate will time out at 773.696593s 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e17fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f6e1809ac40 0x7f6e180b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60366/0 (socket says 172.21.15.136:60366) 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e17fff640 1 -- 172.21.15.136:0/146250249 learned_addr learned my addr 172.21.15.136:0/146250249 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.376+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688637190 0 0) 0x7f6e1809da30 con 0x7f6e1809ac40 2024-03-29T10:34:16.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e00003520 con 0x7f6e1809ac40 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505045368 0 0) 0x7f6e180b39c0 con 0x7f6e180b4360 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1809da30 con 0x7f6e180b4360 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677915470 0 0) 0x7f6e08001110 con 0x7f6e1809c870 2024-03-29T10:34:16.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e180b39c0 con 0x7f6e1809c870 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877039725 0 0) 0x7f6e00003520 con 0x7f6e1809ac40 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 >> v1:172.21.15.117:6790/0 conn(0x7f6e180b4360 legacy=0x7f6e1814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 >> v1:172.21.15.117:6789/0 conn(0x7f6e1809c870 legacy=0x7f6e1813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 monclient: found mon.noname-c 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e180b5160 con 0x7f6e1809ac40 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e0c001690 con 0x7f6e1809ac40 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e1e814640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: dump: 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.375 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e1e814640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e0c001870 con 0x7f6e1809ac40 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 1 -- 172.21.15.136:0/146250249 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e0c001c80 con 0x7f6e1809ac40 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e1e814640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e1e814640 4 set_mon_vals no callback set 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e16ffd640 10 monclient: dump: 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e1e814640 10 monclient: shutdown 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e1e814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e1e814640 1 -- 172.21.15.136:0/146250249 >> v1:172.21.15.136:6789/0 conn(0x7f6e1809ac40 legacy=0x7f6e180b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e1e814640 1 -- 172.21.15.136:0/146250249 shutdown_connections 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.377+0000 7f6e1e814640 1 -- 172.21.15.136:0/146250249 >> 172.21.15.136:0/146250249 conn(0x7f6e18066b00 msgr2=0x7f6e18066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.378+0000 7f6e1e814640 1 -- 172.21.15.136:0/146250249 shutdown_connections 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.378+0000 7f6e1e814640 1 -- 172.21.15.136:0/146250249 wait complete. 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.378+0000 7f6e1e814640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.378+0000 7f6e1e814640 10 monclient: monmap: 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:16.378 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.378+0000 7f6e1e814640 1 librados: starting msgr at 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.378+0000 7f6e1e814640 1 librados: starting objecter 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.378+0000 7f6e1e814640 1 Processor -- start 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 1 -- start start 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 1 librados: setting wanted keys 2024-03-29T10:34:16.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 1 librados: calling monclient init 2024-03-29T10:34:16.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 10 monclient: init 2024-03-29T10:34:16.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 10 monclient(hunting): picked mon.noname-b con 0x7f6e1809ac40 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 10 monclient(hunting): picked mon.noname-c con 0x7f6e18138f60 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 10 monclient(hunting): picked mon.noname-a con 0x7f6e180afdc0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e181db610 con 0x7f6e180afdc0 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e18139a00 con 0x7f6e1809ac40 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e180b0230 con 0x7f6e18138f60 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e1e814640 10 monclient(hunting): authenticate will time out at 773.699624s 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e17fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f6e1809ac40 0x7f6e181dac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60380/0 (socket says 172.21.15.136:60380) 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e17fff640 1 -- 172.21.15.136:0/1538244557 learned_addr learned my addr 172.21.15.136:0/1538244557 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e14ff9640 10 client.?.objecter ms_handle_connect 0x7f6e1809ac40 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e14ff9640 10 client.?.objecter ms_handle_connect 0x7f6e18138f60 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e14ff9640 10 client.?.objecter ms_handle_connect 0x7f6e180afdc0 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.379+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058887906 0 0) 0x7f6e18139a00 con 0x7f6e1809ac40 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e04004eb0 con 0x7f6e1809ac40 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393481817 0 0) 0x7f6e180b0230 con 0x7f6e18138f60 2024-03-29T10:34:16.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e18139a00 con 0x7f6e18138f60 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159189855 0 0) 0x7f6e181db610 con 0x7f6e180afdc0 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e180b0230 con 0x7f6e180afdc0 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2560986539 0 0) 0x7f6e04004eb0 con 0x7f6e1809ac40 2024-03-29T10:34:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e181db610 con 0x7f6e1809ac40 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e0c001bc0 con 0x7f6e1809ac40 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3184939535 0 0) 0x7f6e18139a00 con 0x7f6e18138f60 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e04004eb0 con 0x7f6e18138f60 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e08001860 con 0x7f6e18138f60 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1621363390 0 0) 0x7f6e180b0230 con 0x7f6e180afdc0 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e18139a00 con 0x7f6e180afdc0 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e10001a70 con 0x7f6e180afdc0 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 818417782 0 0) 0x7f6e181db610 con 0x7f6e1809ac40 2024-03-29T10:34:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 >> v1:172.21.15.117:6790/0 conn(0x7f6e18138f60 legacy=0x7f6e180af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 >> v1:172.21.15.117:6789/0 conn(0x7f6e180afdc0 legacy=0x7f6e181e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 monclient: found mon.b 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e180b03c0 con 0x7f6e1809ac40 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e1e814640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e1e814640 10 monclient: _renew_subs 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e1e814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e1e814640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e181e36e0 con 0x7f6e1809ac40 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e1e814640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e1e814640 10 monclient: _renew_subs 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e1e814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e1e814640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e181e3a50 con 0x7f6e1809ac40 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e1e814640 1 librados: init done 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e0c004bf0 con 0x7f6e1809ac40 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e0c004d60 con 0x7f6e1809ac40 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 monclient: dump: 2024-03-29T10:34:16.390 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e167fc640 4 set_mon_vals no callback set 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6e0c005250 con 0x7f6e1809ac40 2024-03-29T10:34:16.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1210671982 0 0) 0x7f6e0c001170 con 0x7f6e1809ac40 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 10 client.4138.objecter ms_dispatch 0x7f6e18000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.380+0000 7f6e14ff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.381+0000 7f6e1e814640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.381+0000 7f6e1e814640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.381+0000 7f6e1e814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.381+0000 7f6e1e814640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df8005080 con 0x7f6e1809ac40 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.385+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6e0c005b90 con 0x7f6e1809ac40 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.385+0000 7f6e14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.385+0000 7f6e14ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:16.401 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.416+0000 7f2e34034040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:34:16.411 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.426+0000 7f414a1ed040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:34:16.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.456+0000 7f6e1e814640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "31b056b7-6332-4aa0-a4da-21f1f5c7f410", "id": 4}] 2024-03-29T10:34:16.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.456+0000 7f6e1e814640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "31b056b7-6332-4aa0-a4da-21f1f5c7f410", "id": 4}] 2024-03-29T10:34:16.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.456+0000 7f6e1e814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.456+0000 7f6e1e814640 1 -- 172.21.15.136:0/1538244557 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "31b056b7-6332-4aa0-a4da-21f1f5c7f410", "id": 4} v 0) v1 -- 0x7f6df80058b0 con 0x7f6e1809ac40 2024-03-29T10:34:16.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.462+0000 7f6e14ff9640 1 -- 172.21.15.136:0/1538244557 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "31b056b7-6332-4aa0-a4da-21f1f5c7f410", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 333574829 0 1697012583) 0x7f6e0c005500 con 0x7f6e1809ac40 2024-03-29T10:34:16.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.462+0000 7f6e14ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "31b056b7-6332-4aa0-a4da-21f1f5c7f410", "id": 4}] 2024-03-29T10:34:16.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.462+0000 7f6e14ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:16.447 INFO:teuthology.orchestra.run.smithi136.stdout:4 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.462+0000 7f6e1e814640 10 librados: watch_flush enter 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.462+0000 7f6e1e814640 10 librados: watch_flush exit 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.462+0000 7f6e1e814640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.462+0000 7f6e1e814640 10 client.4138.objecter successfully canceled tick 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.463+0000 7f6e1e814640 10 monclient: shutdown 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.463+0000 7f6e1e814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.463+0000 7f6e1e814640 1 -- 172.21.15.136:0/1538244557 >> v1:172.21.15.136:6789/0 conn(0x7f6e1809ac40 legacy=0x7f6e181dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.463+0000 7f6e1e814640 1 -- 172.21.15.136:0/1538244557 shutdown_connections 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.463+0000 7f6e1e814640 1 -- 172.21.15.136:0/1538244557 >> 172.21.15.136:0/1538244557 conn(0x7f6e18066b00 msgr2=0x7f6e180b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.463+0000 7f6e1e814640 1 -- 172.21.15.136:0/1538244557 shutdown_connections 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.463+0000 7f6e1e814640 1 -- 172.21.15.136:0/1538244557 wait complete. 2024-03-29T10:34:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.463+0000 7f6e1e814640 1 librados: shutdown 2024-03-29T10:34:16.450 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.463+0000 7f2e34034040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:34:16.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new c98650f7-59e7-4f96-af72-d1369ed68c79 5 2024-03-29T10:34:16.464 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.479+0000 7f414a1ed040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:34:16.505 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.520+0000 7f2e34034040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:34:16.510 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.525+0000 7f414a1ed040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:34:16.554 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.569+0000 7f2e34034040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:34:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.573+0000 7f99415ed640 10 monclient: get_monmap_and_config 2024-03-29T10:34:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.573+0000 7f99415ed640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.573+0000 7f99415ed640 10 monclient: monmap: 2024-03-29T10:34:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:34:16.574037+0000 2024-03-29T10:34:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:34:16.574037+0000 2024-03-29T10:34:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.559 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.559 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:16.559 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:16.559 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.573+0000 7f99415ed640 1 Processor -- start 2024-03-29T10:34:16.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.573+0000 7f99415ed640 1 -- start start 2024-03-29T10:34:16.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.573+0000 7f99415ed640 10 monclient: init 2024-03-29T10:34:16.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:34:16.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 10 monclient(hunting): picked mon.noname-a con 0x7f993c0b1780 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 10 monclient(hunting): picked mon.noname-b con 0x7f993c13d7e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 10 monclient(hunting): picked mon.noname-c con 0x7f993c143be0 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c070b30 con 0x7f993c0b1780 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c070da0 con 0x7f993c143be0 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c09adb0 con 0x7f993c13d7e0 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f99415ed640 10 monclient(hunting): authenticate will time out at 773.894353s 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f993b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f993c143be0 0x7f993c14b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60392/0 (socket says 172.21.15.136:60392) 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f993b7fe640 1 -- 172.21.15.136:0/1630007263 learned_addr learned my addr 172.21.15.136:0/1630007263 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092183165 0 0) 0x7f993c070da0 con 0x7f993c143be0 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9924003520 con 0x7f993c143be0 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197907091 0 0) 0x7f993c09adb0 con 0x7f993c13d7e0 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993c070da0 con 0x7f993c13d7e0 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679427243 0 0) 0x7f993c070b30 con 0x7f993c0b1780 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993c09adb0 con 0x7f993c0b1780 2024-03-29T10:34:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.574+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1598156276 0 0) 0x7f9924003520 con 0x7f993c143be0 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 >> v1:172.21.15.117:6790/0 conn(0x7f993c13d7e0 legacy=0x7f993c1434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 >> v1:172.21.15.117:6789/0 conn(0x7f993c0b1780 legacy=0x7f993c13d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 1 monclient: found mon.noname-c 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c144070 con 0x7f993c143be0 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f9934001a10 con 0x7f993c143be0 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:34:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: dump: 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9934001bf0 con 0x7f993c143be0 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 1 -- 172.21.15.136:0/1630007263 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f9934002000 con 0x7f993c143be0 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 4 set_mon_vals no callback set 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f9939ffb640 10 monclient: dump: 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.572 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 10 monclient: shutdown 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 1 -- 172.21.15.136:0/1630007263 >> v1:172.21.15.136:6789/0 conn(0x7f993c143be0 legacy=0x7f993c14b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 1 -- 172.21.15.136:0/1630007263 shutdown_connections 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 1 -- 172.21.15.136:0/1630007263 >> 172.21.15.136:0/1630007263 conn(0x7f993c0b58f0 msgr2=0x7f993c0b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 1 -- 172.21.15.136:0/1630007263 shutdown_connections 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.575+0000 7f99415ed640 1 -- 172.21.15.136:0/1630007263 wait complete. 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.576+0000 7f99415ed640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.576+0000 7f99415ed640 10 monclient: monmap: 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:16.573 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.576+0000 7f99415ed640 1 librados: starting msgr at 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.576+0000 7f99415ed640 1 librados: starting objecter 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.576+0000 7f99415ed640 1 Processor -- start 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.576+0000 7f99415ed640 1 -- start start 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.576+0000 7f99415ed640 1 librados: setting wanted keys 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.576+0000 7f99415ed640 1 librados: calling monclient init 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.576+0000 7f99415ed640 10 monclient: init 2024-03-29T10:34:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 10 monclient(hunting): picked mon.noname-b con 0x7f993c0b1780 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 10 monclient(hunting): picked mon.noname-a con 0x7f993c13d7e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 10 monclient(hunting): picked mon.noname-c con 0x7f993c143be0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c135660 con 0x7f993c13d7e0 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c143490 con 0x7f993c0b1780 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993c141580 con 0x7f993c143be0 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99415ed640 10 monclient(hunting): authenticate will time out at 773.897465s 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f993affd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f993c0b1780 0x7f993c141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60400/0 (socket says 172.21.15.136:60400) 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f993affd640 1 -- 172.21.15.136:0/2827140921 learned_addr learned my addr 172.21.15.136:0/2827140921 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 client.?.objecter ms_handle_connect 0x7f993c0b1780 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 client.?.objecter ms_handle_connect 0x7f993c13d7e0 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 client.?.objecter ms_handle_connect 0x7f993c143be0 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219771926 0 0) 0x7f993c143490 con 0x7f993c0b1780 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9928004eb0 con 0x7f993c0b1780 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354653400 0 0) 0x7f993c141580 con 0x7f993c143be0 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993c143490 con 0x7f993c143be0 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549847772 0 0) 0x7f993c135660 con 0x7f993c13d7e0 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.577+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993c141580 con 0x7f993c13d7e0 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077497557 0 0) 0x7f993c143490 con 0x7f993c143be0 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f993c135660 con 0x7f993c143be0 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2211382104 0 0) 0x7f9928004eb0 con 0x7f993c0b1780 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f993c143490 con 0x7f993c0b1780 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f9934001f00 con 0x7f993c143be0 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): dump: 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f992c001980 con 0x7f993c0b1780 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): dump: 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 238168637 0 0) 0x7f993c141580 con 0x7f993c13d7e0 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9928004eb0 con 0x7f993c13d7e0 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f99300018b0 con 0x7f993c13d7e0 2024-03-29T10:34:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient(hunting): dump: 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4013513831 0 0) 0x7f993c135660 con 0x7f993c143be0 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 >> v1:172.21.15.136:6789/0 conn(0x7f993c0b1780 legacy=0x7f993c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 >> v1:172.21.15.117:6789/0 conn(0x7f993c13d7e0 legacy=0x7f993c134840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 monclient: found mon.c 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c1421b0 con 0x7f993c143be0 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99415ed640 5 monclient: authenticate success, global_id 4151 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99415ed640 10 monclient: _renew_subs 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99415ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99415ed640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993c1424c0 con 0x7f993c143be0 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99415ed640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99415ed640 10 monclient: _renew_subs 2024-03-29T10:34:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99415ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99415ed640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f993c06ad40 con 0x7f993c143be0 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99415ed640 1 librados: init done 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9934001420 con 0x7f993c143be0 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f9934004fb0 con 0x7f993c143be0 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99397fa640 4 set_mon_vals no callback set 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 monclient: dump: 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.582 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f99340054a0 con 0x7f993c143be0 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3824760882 0 0) 0x7f9934005cd0 con 0x7f993c143be0 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 10 client.4151.objecter ms_dispatch 0x7f993c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 3 client.4151.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 3 client.4151.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.578+0000 7f99237fe640 20 client.4151.objecter dump_active .. 0 homeless 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.579+0000 7f99415ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.579+0000 7f99415ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.579+0000 7f99415ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.579+0000 7f99415ed640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c005080 con 0x7f993c143be0 2024-03-29T10:34:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.581+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9934005fb0 con 0x7f993c143be0 2024-03-29T10:34:16.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.581+0000 7f99237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.581+0000 7f99237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:16.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.652+0000 7f99415ed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c98650f7-59e7-4f96-af72-d1369ed68c79", "id": 5}] 2024-03-29T10:34:16.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.652+0000 7f99415ed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c98650f7-59e7-4f96-af72-d1369ed68c79", "id": 5}] 2024-03-29T10:34:16.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.652+0000 7f99415ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.652+0000 7f99415ed640 1 -- 172.21.15.136:0/2827140921 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd new", "uuid": "c98650f7-59e7-4f96-af72-d1369ed68c79", "id": 5} v 0) v1 -- 0x7f991c0058b0 con 0x7f993c143be0 2024-03-29T10:34:16.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.655+0000 7f99237fe640 1 -- 172.21.15.136:0/2827140921 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c98650f7-59e7-4f96-af72-d1369ed68c79", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3922285056 0 1988412176) 0x7f9934005750 con 0x7f993c143be0 2024-03-29T10:34:16.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.655+0000 7f99237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c98650f7-59e7-4f96-af72-d1369ed68c79", "id": 5}] 2024-03-29T10:34:16.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.655+0000 7f99237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:16.641 INFO:teuthology.orchestra.run.smithi136.stdout:5 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 10 librados: watch_flush enter 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 10 librados: watch_flush exit 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 20 client.4151.objecter shutdown clearing up homeless session... 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 10 client.4151.objecter successfully canceled tick 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 10 monclient: shutdown 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 1 -- 172.21.15.136:0/2827140921 >> v1:172.21.15.117:6790/0 conn(0x7f993c143be0 legacy=0x7f993c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 1 -- 172.21.15.136:0/2827140921 shutdown_connections 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 1 -- 172.21.15.136:0/2827140921 >> 172.21.15.136:0/2827140921 conn(0x7f993c0b58f0 msgr2=0x7f993c069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 1 -- 172.21.15.136:0/2827140921 shutdown_connections 2024-03-29T10:34:16.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 1 -- 172.21.15.136:0/2827140921 wait complete. 2024-03-29T10:34:16.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.656+0000 7f99415ed640 1 librados: shutdown 2024-03-29T10:34:16.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 54aeac70-ee32-4acc-ba23-b1ca8f788b88 6 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.765+0000 7f5da5cd9640 10 monclient: get_monmap_and_config 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.765+0000 7f5da5cd9640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.765+0000 7f5da5cd9640 10 monclient: monmap: 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:34:16.766673+0000 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:34:16.766673+0000 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:16.751 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.766+0000 7f5da5cd9640 1 Processor -- start 2024-03-29T10:34:16.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.766+0000 7f5da5cd9640 1 -- start start 2024-03-29T10:34:16.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.766+0000 7f5da5cd9640 10 monclient: init 2024-03-29T10:34:16.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:16.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f5da009ac40 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f5da009c870 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f5da00b4360 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da009dec0 con 0x7f5da00b4360 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da009da30 con 0x7f5da009ac40 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da00b39c0 con 0x7f5da009c870 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5da5cd9640 10 monclient(hunting): authenticate will time out at 774.087320s 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5da009ac40 0x7f5da00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60402/0 (socket says 172.21.15.136:60402) 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9f7fe640 1 -- 172.21.15.136:0/2923581894 learned_addr learned my addr 172.21.15.136:0/2923581894 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137724648 0 0) 0x7f5da009da30 con 0x7f5da009ac40 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c003520 con 0x7f5da009ac40 2024-03-29T10:34:16.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150644157 0 0) 0x7f5d98001490 con 0x7f5da00b4360 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da009da30 con 0x7f5da00b4360 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149318035 0 0) 0x7f5da00b39c0 con 0x7f5da009c870 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.767+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d98001490 con 0x7f5da009c870 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626696617 0 0) 0x7f5d8c003520 con 0x7f5da009ac40 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 >> v1:172.21.15.117:6790/0 conn(0x7f5da009c870 legacy=0x7f5da013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 >> v1:172.21.15.117:6789/0 conn(0x7f5da00b4360 legacy=0x7f5da014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 1 monclient: found mon.noname-c 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da00b5160 con 0x7f5da009ac40 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5d90001690 con 0x7f5da009ac40 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: dump: 2024-03-29T10:34:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d900017f0 con 0x7f5da009ac40 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 4 set_mon_vals no callback set 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 1 -- 172.21.15.136:0/2923581894 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5d90001c00 con 0x7f5da009ac40 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5d9effd640 10 monclient: dump: 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 10 monclient: shutdown 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 1 -- 172.21.15.136:0/2923581894 >> v1:172.21.15.136:6789/0 conn(0x7f5da009ac40 legacy=0x7f5da00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 1 -- 172.21.15.136:0/2923581894 shutdown_connections 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 1 -- 172.21.15.136:0/2923581894 >> 172.21.15.136:0/2923581894 conn(0x7f5da0066b00 msgr2=0x7f5da0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 1 -- 172.21.15.136:0/2923581894 shutdown_connections 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.768+0000 7f5da5cd9640 1 -- 172.21.15.136:0/2923581894 wait complete. 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.769+0000 7f5da5cd9640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.769+0000 7f5da5cd9640 10 monclient: monmap: 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.766 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.769+0000 7f5da5cd9640 1 librados: starting msgr at 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.769+0000 7f5da5cd9640 1 librados: starting objecter 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.769+0000 7f5da5cd9640 1 Processor -- start 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.769+0000 7f5da5cd9640 1 -- start start 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.769+0000 7f5da5cd9640 1 librados: setting wanted keys 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.769+0000 7f5da5cd9640 1 librados: calling monclient init 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.769+0000 7f5da5cd9640 10 monclient: init 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f5da009ac40 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f5da01c88e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f5da01cacb0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da01ca2b0 con 0x7f5da009ac40 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da01ca520 con 0x7f5da01c88e0 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da01c85c0 con 0x7f5da01cacb0 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5da5cd9640 10 monclient(hunting): authenticate will time out at 774.090360s 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d97fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5da01c88e0 0x7f5da00b1000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60414/0 (socket says 172.21.15.136:60414) 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d97fff640 1 -- 172.21.15.136:0/1385890709 learned_addr learned my addr 172.21.15.136:0/1385890709 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 client.?.objecter ms_handle_connect 0x7f5da01c88e0 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 client.?.objecter ms_handle_connect 0x7f5da009ac40 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 client.?.objecter ms_handle_connect 0x7f5da01cacb0 2024-03-29T10:34:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182611130 0 0) 0x7f5da01ca520 con 0x7f5da01c88e0 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d80004eb0 con 0x7f5da01c88e0 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054350899 0 0) 0x7f5da01c85c0 con 0x7f5da01cacb0 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): my global_id is 4157 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da01ca520 con 0x7f5da01cacb0 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582560039 0 0) 0x7f5da01ca2b0 con 0x7f5da009ac40 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da01c85c0 con 0x7f5da009ac40 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4231715673 0 0) 0x7f5d80004eb0 con 0x7f5da01c88e0 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5da01ca2b0 con 0x7f5da01c88e0 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5d88001a10 con 0x7f5da01c88e0 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): dump: 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.770 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.770+0000 7f5d9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3099513800 0 0) 0x7f5da01ca520 con 0x7f5da01cacb0 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d80004eb0 con 0x7f5da01cacb0 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5d98001b40 con 0x7f5da01cacb0 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient(hunting): dump: 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.771 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3384967092 0 0) 0x7f5da01c85c0 con 0x7f5da009ac40 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5da01ca520 con 0x7f5da009ac40 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5d90001b00 con 0x7f5da009ac40 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient(hunting): dump: 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.772 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 944600062 0 0) 0x7f5da01ca2b0 con 0x7f5da01c88e0 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 >> v1:172.21.15.117:6790/0 conn(0x7f5da01cacb0 legacy=0x7f5da01e2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 >> v1:172.21.15.117:6789/0 conn(0x7f5da009ac40 legacy=0x7f5da01d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 monclient: found mon.b 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da00b1710 con 0x7f5da01c88e0 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5da5cd9640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5da5cd9640 10 monclient: _renew_subs 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5da5cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5da5cd9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da00b1960 con 0x7f5da01c88e0 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5da5cd9640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5da5cd9640 10 monclient: _renew_subs 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5da5cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5da5cd9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da0142d50 con 0x7f5da01c88e0 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5da5cd9640 1 librados: init done 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d88001c70 con 0x7f5da01c88e0 2024-03-29T10:34:16.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5d88003a60 con 0x7f5da01c88e0 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9e7fc640 4 set_mon_vals no callback set 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 monclient: dump: 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5d88003fd0 con 0x7f5da01c88e0 2024-03-29T10:34:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 903704169 0 0) 0x7f5d88005db0 con 0x7f5da01c88e0 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 10 client.4150.objecter ms_dispatch 0x7f5da0000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.771+0000 7f5d9cff9640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.772+0000 7f5da5cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.772+0000 7f5da5cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.772+0000 7f5da5cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.772+0000 7f5da5cd9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d84005080 con 0x7f5da01c88e0 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.775+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5d880042e0 con 0x7f5da01c88e0 2024-03-29T10:34:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.775+0000 7f5d9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.775+0000 7f5d9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:16.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.848+0000 7f5da5cd9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "54aeac70-ee32-4acc-ba23-b1ca8f788b88", "id": 6}] 2024-03-29T10:34:16.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.848+0000 7f5da5cd9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "54aeac70-ee32-4acc-ba23-b1ca8f788b88", "id": 6}] 2024-03-29T10:34:16.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.848+0000 7f5da5cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.848+0000 7f5da5cd9640 1 -- 172.21.15.136:0/1385890709 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "54aeac70-ee32-4acc-ba23-b1ca8f788b88", "id": 6} v 0) v1 -- 0x7f5d840058b0 con 0x7f5da01c88e0 2024-03-29T10:34:16.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.856+0000 7f5d9cff9640 1 -- 172.21.15.136:0/1385890709 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "54aeac70-ee32-4acc-ba23-b1ca8f788b88", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 465075332 0 1113811849) 0x7f5d880057c0 con 0x7f5da01c88e0 2024-03-29T10:34:16.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.856+0000 7f5d9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "54aeac70-ee32-4acc-ba23-b1ca8f788b88", "id": 6}] 2024-03-29T10:34:16.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.856+0000 7f5d9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:16.841 INFO:teuthology.orchestra.run.smithi136.stdout:6 2024-03-29T10:34:16.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 10 librados: watch_flush enter 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 10 librados: watch_flush exit 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 10 client.4150.objecter successfully canceled tick 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 10 monclient: shutdown 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 1 -- 172.21.15.136:0/1385890709 >> v1:172.21.15.136:6789/0 conn(0x7f5da01c88e0 legacy=0x7f5da00b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 1 -- 172.21.15.136:0/1385890709 shutdown_connections 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 1 -- 172.21.15.136:0/1385890709 >> 172.21.15.136:0/1385890709 conn(0x7f5da0066b00 msgr2=0x7f5da00b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 1 -- 172.21.15.136:0/1385890709 shutdown_connections 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 1 -- 172.21.15.136:0/1385890709 wait complete. 2024-03-29T10:34:16.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.857+0000 7f5da5cd9640 1 librados: shutdown 2024-03-29T10:34:16.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 6f111ef1-b009-4406-abef-7aca27f83384 7 2024-03-29T10:34:16.893 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:16.908+0000 7f414a1ed040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:34:16.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.967+0000 7f258364e640 10 monclient: get_monmap_and_config 2024-03-29T10:34:16.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.967+0000 7f258364e640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.967+0000 7f258364e640 10 monclient: monmap: 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:34:16.968169+0000 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:34:16.968169+0000 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.967+0000 7f258364e640 1 Processor -- start 2024-03-29T10:34:16.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 1 -- start start 2024-03-29T10:34:16.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 10 monclient: init 2024-03-29T10:34:16.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:34:16.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 10 monclient(hunting): picked mon.noname-c con 0x7f257c09ac40 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 10 monclient(hunting): picked mon.noname-a con 0x7f257c09c870 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 10 monclient(hunting): picked mon.noname-b con 0x7f257c0b4360 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c09dec0 con 0x7f257c09c870 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c09da30 con 0x7f257c09ac40 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0b39c0 con 0x7f257c0b4360 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f258364e640 10 monclient(hunting): authenticate will time out at 774.288678s 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f25813ea640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f257c09ac40 0x7f257c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60426/0 (socket says 172.21.15.136:60426) 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.968+0000 7f25813ea640 1 -- 172.21.15.136:0/84997816 learned_addr learned my addr 172.21.15.136:0/84997816 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59865723 0 0) 0x7f257c09da30 con 0x7f257c09ac40 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2568003520 con 0x7f257c09ac40 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454637286 0 0) 0x7f257c0b39c0 con 0x7f257c0b4360 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): my global_id is 4160 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f257c09da30 con 0x7f257c0b4360 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076102105 0 0) 0x7f256c001110 con 0x7f257c09c870 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f257c0b39c0 con 0x7f257c09c870 2024-03-29T10:34:16.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 689497577 0 0) 0x7f2568003520 con 0x7f257c09ac40 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 >> v1:172.21.15.117:6790/0 conn(0x7f257c0b4360 legacy=0x7f257c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 >> v1:172.21.15.117:6789/0 conn(0x7f257c09c870 legacy=0x7f257c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 monclient: found mon.noname-c 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0b5160 con 0x7f257c09ac40 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2578001690 con 0x7f257c09ac40 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f258364e640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:34:16.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f258364e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: dump: 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f258364e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2578001870 con 0x7f257c09ac40 2024-03-29T10:34:16.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 1 -- 172.21.15.136:0/84997816 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2578001c80 con 0x7f257c09ac40 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f258364e640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f258364e640 4 set_mon_vals no callback set 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.969+0000 7f2573fff640 10 monclient: dump: 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.966 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.970+0000 7f258364e640 10 monclient: shutdown 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.970+0000 7f258364e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.970+0000 7f258364e640 1 -- 172.21.15.136:0/84997816 >> v1:172.21.15.136:6789/0 conn(0x7f257c09ac40 legacy=0x7f257c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.970+0000 7f258364e640 1 -- 172.21.15.136:0/84997816 shutdown_connections 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.970+0000 7f258364e640 1 -- 172.21.15.136:0/84997816 >> 172.21.15.136:0/84997816 conn(0x7f257c066b00 msgr2=0x7f257c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.970+0000 7f258364e640 1 -- 172.21.15.136:0/84997816 shutdown_connections 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.970+0000 7f258364e640 1 -- 172.21.15.136:0/84997816 wait complete. 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.970+0000 7f258364e640 10 monclient: build_initial_monmap 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.970+0000 7f258364e640 10 monclient: monmap: 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:16.967 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 1 librados: starting msgr at 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 1 librados: starting objecter 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 1 Processor -- start 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 1 -- start start 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 1 librados: setting wanted keys 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 1 librados: calling monclient init 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 10 monclient: init 2024-03-29T10:34:16.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 10 monclient(hunting): picked mon.noname-c con 0x7f257c09ac40 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 10 monclient(hunting): picked mon.noname-a con 0x7f257c138f60 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 10 monclient(hunting): picked mon.noname-b con 0x7f257c0afdc0 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c1db610 con 0x7f257c138f60 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c139a00 con 0x7f257c0afdc0 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0b0230 con 0x7f257c09ac40 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f258364e640 10 monclient(hunting): authenticate will time out at 774.291806s 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f2581beb640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f257c0afdc0 0x7f257c1e1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60442/0 (socket says 172.21.15.136:60442) 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.971+0000 7f2581beb640 1 -- 172.21.15.136:0/2838296844 learned_addr learned my addr 172.21.15.136:0/2838296844 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 client.?.objecter ms_handle_connect 0x7f257c0afdc0 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 client.?.objecter ms_handle_connect 0x7f257c09ac40 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 client.?.objecter ms_handle_connect 0x7f257c138f60 2024-03-29T10:34:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839220812 0 0) 0x7f257c139a00 con 0x7f257c0afdc0 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): my global_id is 4156 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2560004eb0 con 0x7f257c0afdc0 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136203001 0 0) 0x7f257c0b0230 con 0x7f257c09ac40 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): my global_id is 4163 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f257c139a00 con 0x7f257c09ac40 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577994315 0 0) 0x7f257c1db610 con 0x7f257c138f60 2024-03-29T10:34:16.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): my global_id is 4161 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f257c0b0230 con 0x7f257c138f60 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 590281727 0 0) 0x7f257c139a00 con 0x7f257c09ac40 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f257c1db610 con 0x7f257c09ac40 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2578001b80 con 0x7f257c09ac40 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): dump: 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.971 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3874047237 0 0) 0x7f2560004eb0 con 0x7f257c0afdc0 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f257c139a00 con 0x7f257c0afdc0 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2574001bd0 con 0x7f257c0afdc0 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): dump: 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.972 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2251145782 0 0) 0x7f257c0b0230 con 0x7f257c138f60 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2560004eb0 con 0x7f257c138f60 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f256c001760 con 0x7f257c138f60 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient(hunting): dump: 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.973 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3886902607 0 0) 0x7f257c1db610 con 0x7f257c09ac40 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 >> v1:172.21.15.136:6789/0 conn(0x7f257c0afdc0 legacy=0x7f257c1e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 >> v1:172.21.15.117:6789/0 conn(0x7f257c138f60 legacy=0x7f257c0af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 monclient: found mon.c 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:16.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0b03c0 con 0x7f257c09ac40 2024-03-29T10:34:16.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient: _finish_auth 0 2024-03-29T10:34:16.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:34:16.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:16.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f258364e640 5 monclient: authenticate success, global_id 4163 2024-03-29T10:34:16.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f258364e640 10 monclient: _renew_subs 2024-03-29T10:34:16.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f258364e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f258364e640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c1e36e0 con 0x7f257c09ac40 2024-03-29T10:34:16.975 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2e34034040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2578004bf0 con 0x7f257c09ac40 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2578004d60 con 0x7f257c09ac40 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 10 monclient: dump: 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f25737fe640 4 set_mon_vals no callback set 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f258364e640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f258364e640 10 monclient: _renew_subs 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f258364e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f258364e640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c1e3a50 con 0x7f257c09ac40 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f258364e640 1 librados: init done 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.972+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2578005250 con 0x7f257c09ac40 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.973+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 4190108936 0 0) 0x7f2578005b60 con 0x7f257c09ac40 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.973+0000 7f2571ffb640 10 client.4163.objecter ms_dispatch 0x7f257c000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.973+0000 7f2571ffb640 3 client.4163.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:34:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.973+0000 7f2571ffb640 3 client.4163.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:34:16.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.973+0000 7f2571ffb640 20 client.4163.objecter dump_active .. 0 homeless 2024-03-29T10:34:16.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.973+0000 7f258364e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.973+0000 7f258364e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.973+0000 7f258364e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:16.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.973+0000 7f258364e640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c1db3f0 con 0x7f257c09ac40 2024-03-29T10:34:16.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.975+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2578005d20 con 0x7f257c09ac40 2024-03-29T10:34:16.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.976+0000 7f2571ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:16.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:16.976+0000 7f2571ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:17.026 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:17.041+0000 7f414a1ed040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:34:17.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.049+0000 7f258364e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6f111ef1-b009-4406-abef-7aca27f83384", "id": 7}] 2024-03-29T10:34:17.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.049+0000 7f258364e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6f111ef1-b009-4406-abef-7aca27f83384", "id": 7}] 2024-03-29T10:34:17.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.049+0000 7f258364e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:17.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.049+0000 7f258364e640 1 -- 172.21.15.136:0/2838296844 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd new", "uuid": "6f111ef1-b009-4406-abef-7aca27f83384", "id": 7} v 0) v1 -- 0x7f257c04f730 con 0x7f257c09ac40 2024-03-29T10:34:17.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.053+0000 7f2571ffb640 1 -- 172.21.15.136:0/2838296844 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6f111ef1-b009-4406-abef-7aca27f83384", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 894499100 0 1371665406) 0x7f2578005ec0 con 0x7f257c09ac40 2024-03-29T10:34:17.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.053+0000 7f2571ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6f111ef1-b009-4406-abef-7aca27f83384", "id": 7}] 2024-03-29T10:34:17.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.053+0000 7f2571ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:17.038 INFO:teuthology.orchestra.run.smithi136.stdout:7 2024-03-29T10:34:17.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.053+0000 7f258364e640 10 librados: watch_flush enter 2024-03-29T10:34:17.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.053+0000 7f258364e640 10 librados: watch_flush exit 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.053+0000 7f258364e640 20 client.4163.objecter shutdown clearing up homeless session... 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.053+0000 7f258364e640 10 client.4163.objecter successfully canceled tick 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.054+0000 7f258364e640 10 monclient: shutdown 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.054+0000 7f258364e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.054+0000 7f258364e640 1 -- 172.21.15.136:0/2838296844 >> v1:172.21.15.117:6790/0 conn(0x7f257c09ac40 legacy=0x7f257c1dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.054+0000 7f258364e640 1 -- 172.21.15.136:0/2838296844 shutdown_connections 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.054+0000 7f258364e640 1 -- 172.21.15.136:0/2838296844 >> 172.21.15.136:0/2838296844 conn(0x7f257c066b00 msgr2=0x7f257c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.054+0000 7f258364e640 1 -- 172.21.15.136:0/2838296844 shutdown_connections 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.054+0000 7f258364e640 1 -- 172.21.15.136:0/2838296844 wait complete. 2024-03-29T10:34:17.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-29T10:34:17.054+0000 7f258364e640 1 librados: shutdown 2024-03-29T10:34:17.049 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:34:17.049 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:34:17.051 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:34:17.051 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:34:17.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:34:17.053 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:34:17.053 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:34:17.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:34:17.055 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:34:17.055 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:34:17.055 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:34:17.057 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:34:17.057 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:34:17.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:34:17.059 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:34:17.059 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:34:17.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:34:17.061 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:34:17.062 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:34:17.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:34:17.063 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:34:17.064 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:34:17.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:34:17.066 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:34:17.066 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:34:17.106 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:17.121+0000 7f2e34034040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:34:17.164 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:17.179+0000 7f414a1ed040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:34:17.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.205+0000 7f816fe25640 10 monclient: get_monmap_and_config 2024-03-29T10:34:17.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.205+0000 7f816fe25640 10 monclient: build_initial_monmap 2024-03-29T10:34:17.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.205+0000 7f816fe25640 10 monclient: monmap: 2024-03-29T10:34:17.190 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-29T10:34:17.191 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:17.191 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:34:17.206394+0000 2024-03-29T10:34:17.191 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:34:17.206394+0000 2024-03-29T10:34:17.191 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:17.191 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:17.191 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:17.191 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:17.191 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:17.191 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:17.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 1 Processor -- start 2024-03-29T10:34:17.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 1 -- start start 2024-03-29T10:34:17.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 10 monclient: init 2024-03-29T10:34:17.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:17.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 10 monclient(hunting): picked mon.noname-a con 0x7f81680b8240 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 10 monclient(hunting): picked mon.noname-c con 0x7f81680b1f60 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 10 monclient(hunting): picked mon.noname-b con 0x7f816809c8c0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816809bd80 con 0x7f81680b8240 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816809b960 con 0x7f81680b1f60 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81680b2ec0 con 0x7f816809c8c0 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.206+0000 7f816fe25640 10 monclient(hunting): authenticate will time out at 778.469013s 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816e3c2640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f816809c8c0 0x7f816814d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58554/0 (socket says 172.21.15.117:58554) 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816e3c2640 1 -- 172.21.15.117:0/4120893180 learned_addr learned my addr 172.21.15.117:0/4120893180 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43192571 0 0) 0x7f81680b2ec0 con 0x7f816809c8c0 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): _negotiate 2024-03-29T10:34:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): my global_id is 4190 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8154003520 con 0x7f816809c8c0 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184829996 0 0) 0x7f816809bd80 con 0x7f81680b8240 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): _negotiate 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81680b2ec0 con 0x7f81680b8240 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267448470 0 0) 0x7f816809b960 con 0x7f81680b1f60 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): _negotiate 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f816809bd80 con 0x7f81680b1f60 2024-03-29T10:34:17.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9830205 0 0) 0x7f8154003520 con 0x7f816809c8c0 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 >> v1:172.21.15.136:6789/0 conn(0x7f81680b1f60 legacy=0x7f816809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 >> v1:172.21.15.117:6789/0 conn(0x7f81680b8240 legacy=0x7f816813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: _finish_hunting 0 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 monclient: found mon.noname-b 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816813fb60 con 0x7f816809c8c0 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: _finish_auth 0 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: _check_auth_tickets 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8164001690 con 0x7f816809c8c0 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816fe25640 5 monclient: authenticate success, global_id 4190 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816fe25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: dump: 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816fe25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81640017f0 con 0x7f816809c8c0 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 1 -- 172.21.15.117:0/4120893180 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8164001c40 con 0x7f816809c8c0 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816fe25640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:17.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816fe25640 4 set_mon_vals no callback set 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.207+0000 7f816cbbf640 10 monclient: dump: 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.208+0000 7f816fe25640 10 monclient: shutdown 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.208+0000 7f816fe25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:17.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.208+0000 7f816fe25640 1 -- 172.21.15.117:0/4120893180 >> v1:172.21.15.117:6790/0 conn(0x7f816809c8c0 legacy=0x7f816814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.208+0000 7f816fe25640 1 -- 172.21.15.117:0/4120893180 shutdown_connections 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.208+0000 7f816fe25640 1 -- 172.21.15.117:0/4120893180 >> 172.21.15.117:0/4120893180 conn(0x7f8168066b00 msgr2=0x7f8168066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.208+0000 7f816fe25640 1 -- 172.21.15.117:0/4120893180 shutdown_connections 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.208+0000 7f816fe25640 1 -- 172.21.15.117:0/4120893180 wait complete. 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient: build_initial_monmap 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient: monmap: 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:17.207 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 1 librados: starting msgr at 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 1 librados: starting objecter 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 1 Processor -- start 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 1 -- start start 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 1 librados: setting wanted keys 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 1 librados: calling monclient init 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient: init 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient(hunting): picked mon.noname-a con 0x7f816809c8c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient(hunting): picked mon.noname-b con 0x7f81680b8240 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient(hunting): picked mon.noname-c con 0x7f8168143860 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81681d8080 con 0x7f816809c8c0 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8168132f90 con 0x7f81680b8240 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81681d7170 con 0x7f8168143860 2024-03-29T10:34:17.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.209+0000 7f816fe25640 10 monclient(hunting): authenticate will time out at 778.472077s 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816dbc1640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f816809c8c0 0x7f81681316e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:51562/0 (socket says 172.21.15.117:51562) 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816dbc1640 1 -- 172.21.15.117:0/3733183627 learned_addr learned my addr 172.21.15.117:0/3733183627 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 client.?.objecter ms_handle_connect 0x7f816809c8c0 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 client.?.objecter ms_handle_connect 0x7f8168143860 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 client.?.objecter ms_handle_connect 0x7f81680b8240 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686051398 0 0) 0x7f81681d7170 con 0x7f8168143860 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): my global_id is 4193 2024-03-29T10:34:17.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f814c004eb0 con 0x7f8168143860 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156613958 0 0) 0x7f81681d8080 con 0x7f816809c8c0 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81681d7170 con 0x7f816809c8c0 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844622844 0 0) 0x7f8168132f90 con 0x7f81680b8240 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81681d8080 con 0x7f81680b8240 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2780910083 0 0) 0x7f814c004eb0 con 0x7f8168143860 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8168132f90 con 0x7f8168143860 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8164001960 con 0x7f8168143860 2024-03-29T10:34:17.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): dump: 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 609232529 0 0) 0x7f81681d7170 con 0x7f816809c8c0 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f814c004eb0 con 0x7f816809c8c0 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1006170402 0 0) 0x7f81681d8080 con 0x7f81680b8240 2024-03-29T10:34:17.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81681d7170 con 0x7f81680b8240 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8158001830 con 0x7f81680b8240 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): dump: 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8160001cd0 con 0x7f816809c8c0 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:17.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient(hunting): dump: 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1726715060 0 0) 0x7f8168132f90 con 0x7f8168143860 2024-03-29T10:34:17.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 >> v1:172.21.15.136:6789/0 conn(0x7f81680b8240 legacy=0x7f81681d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 >> v1:172.21.15.117:6789/0 conn(0x7f816809c8c0 legacy=0x7f81681316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 monclient: found mon.c 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81681d7450 con 0x7f8168143860 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816fe25640 5 monclient: authenticate success, global_id 4193 2024-03-29T10:34:17.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816fe25640 10 monclient: _renew_subs 2024-03-29T10:34:17.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816fe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:17.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816fe25640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81681460d0 con 0x7f8168143860 2024-03-29T10:34:17.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816fe25640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:17.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816fe25640 10 monclient: _renew_subs 2024-03-29T10:34:17.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816fe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:17.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816fe25640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81681e34b0 con 0x7f8168143860 2024-03-29T10:34:17.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f816fe25640 1 librados: init done 2024-03-29T10:34:17.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8164004940 con 0x7f8168143860 2024-03-29T10:34:17.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8164004c90 con 0x7f8168143860 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815e7fc640 10 monclient: dump: 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.210+0000 7f815ffff640 4 set_mon_vals no callback set 2024-03-29T10:34:17.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8164005240 con 0x7f8168143860 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f81640069b0 con 0x7f8168143860 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f815e7fc640 10 client.4193.objecter ms_dispatch 0x7f8168000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f815e7fc640 3 client.4193.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f815e7fc640 3 client.4193.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f815e7fc640 20 client.4193.objecter dump_active .. 0 homeless 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f816fe25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f816fe25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f816fe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.211+0000 7f816fe25640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8150005080 con 0x7f8168143860 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.213+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8164006bc0 con 0x7f8168143860 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.213+0000 7f815e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:17.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.213+0000 7f815e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:17.218 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:17.227+0000 7f414a1ed040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:34:17.236 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:17.251+0000 7f2e34034040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:34:17.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.274+0000 7f816fe25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:17.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.274+0000 7f816fe25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:17.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.274+0000 7f816fe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:17.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.274+0000 7f816fe25640 1 -- 172.21.15.117:0/3733183627 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8150005330 con 0x7f8168143860 2024-03-29T10:34:17.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.275+0000 7f815e7fc640 1 -- 172.21.15.117:0/3733183627 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f8164006390 con 0x7f8168143860 2024-03-29T10:34:17.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.275+0000 7f815e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:17.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.275+0000 7f815e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:17.260 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T10:34:17.260 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:34:17.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 10 librados: watch_flush enter 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 10 librados: watch_flush exit 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 20 client.4193.objecter shutdown clearing up homeless session... 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 10 client.4193.objecter successfully canceled tick 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 10 monclient: shutdown 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 1 -- 172.21.15.117:0/3733183627 >> v1:172.21.15.117:6790/0 conn(0x7f8168143860 legacy=0x7f81681e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 1 -- 172.21.15.117:0/3733183627 shutdown_connections 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 1 -- 172.21.15.117:0/3733183627 >> 172.21.15.117:0/3733183627 conn(0x7f8168066b00 msgr2=0x7f816806bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 1 -- 172.21.15.117:0/3733183627 shutdown_connections 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 1 -- 172.21.15.117:0/3733183627 wait complete. 2024-03-29T10:34:17.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T10:34:17.276+0000 7f816fe25640 1 librados: shutdown 2024-03-29T10:34:17.271 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:34:17.272 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:34:17.283 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:17.298+0000 7f2e34034040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:34:17.329 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:17.343+0000 7f414a1ed040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:34:17.394 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:17.408+0000 7f2e34034040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:34:17.424 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:17.439+0000 7f414a1ed040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:34:17.491 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:17.506+0000 7f2e34034040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:34:17.576 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:17.591+0000 7f414a1ed040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:34:17.618 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:17.633+0000 7f414a1ed040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:34:17.642 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:17.657+0000 7f2e34034040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:34:17.665 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-03-29T10:34:17.680+0000 7f62c2060640 -1 Falling back to public interface 2024-03-29T10:34:17.666 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-03-29T10:34:17.681+0000 7f424ff26640 -1 Falling back to public interface 2024-03-29T10:34:17.670 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-03-29T10:34:17.685+0000 7f9afab26640 -1 Falling back to public interface 2024-03-29T10:34:17.671 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-03-29T10:34:17.685+0000 7f1eaf721640 -1 Falling back to public interface 2024-03-29T10:34:17.674 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-03-29T10:34:17.689+0000 7fbfec61b640 -1 Falling back to public interface 2024-03-29T10:34:17.677 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-03-29T10:34:17.692+0000 7f1f3521e640 -1 Falling back to public interface 2024-03-29T10:34:17.680 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-03-29T10:34:17.695+0000 7f317d16f640 -1 Falling back to public interface 2024-03-29T10:34:17.681 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-03-29T10:34:17.697+0000 7fcea500d640 -1 Falling back to public interface 2024-03-29T10:34:17.687 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:17.702+0000 7f2e34034040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:34:17.776 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:17.791+0000 7f414a1ed040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:34:17.839 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:17.854+0000 7f2e34034040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:34:18.078 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:34:18.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.184+0000 7f7a85fb9640 10 monclient: get_monmap_and_config 2024-03-29T10:34:18.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.184+0000 7f7a85fb9640 10 monclient: build_initial_monmap 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.184+0000 7f7a85fb9640 10 monclient: monmap: 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:34:18.185387+0000 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:34:18.185387+0000 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:18.170 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:18.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 1 Processor -- start 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 1 -- start start 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 10 monclient: init 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 10 monclient(hunting): picked mon.noname-c con 0x7f7a801457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8013f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 10 monclient(hunting): picked mon.noname-b con 0x7f7a800a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a800aaef0 con 0x7f7a8013f650 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a80145bd0 con 0x7f7a801457f0 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a80145eb0 con 0x7f7a800a9d70 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a85fb9640 10 monclient(hunting): authenticate will time out at 779.447959s 2024-03-29T10:34:18.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a7effd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7a8013f650 0x7f7a800b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:51578/0 (socket says 172.21.15.117:51578) 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a7effd640 1 -- 172.21.15.117:0/3532190945 learned_addr learned my addr 172.21.15.117:0/3532190945 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.185+0000 7f7a7ffff640 1 --1- 172.21.15.117:0/3532190945 >> v1:172.21.15.117:6790/0 conn(0x7f7a800a9d70 0x7f7a8014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58568/0 (socket says 172.21.15.117:58568) 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269469485 0 0) 0x7f7a80145eb0 con 0x7f7a800a9d70 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): my global_id is 4196 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a68003520 con 0x7f7a800a9d70 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478354190 0 0) 0x7f7a800aaef0 con 0x7f7a8013f650 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a80145eb0 con 0x7f7a8013f650 2024-03-29T10:34:18.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540090094 0 0) 0x7f7a80145bd0 con 0x7f7a801457f0 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a800aaef0 con 0x7f7a801457f0 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4167745312 0 0) 0x7f7a80145eb0 con 0x7f7a8013f650 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 >> v1:172.21.15.117:6790/0 conn(0x7f7a800a9d70 legacy=0x7f7a8014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 >> v1:172.21.15.136:6789/0 conn(0x7f7a801457f0 legacy=0x7f7a8013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 monclient: found mon.noname-a 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a80146340 con 0x7f7a8013f650 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:18.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7a74001670 con 0x7f7a8013f650 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a85fb9640 5 monclient: authenticate success, global_id 4194 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a85fb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: dump: 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:18.181 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a85fb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a740017d0 con 0x7f7a8013f650 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 1 -- 172.21.15.117:0/3532190945 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7a74001c20 con 0x7f7a8013f650 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a85fb9640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a85fb9640 4 set_mon_vals no callback set 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.186+0000 7f7a7e7fc640 10 monclient: dump: 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.182 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.187+0000 7f7a85fb9640 10 monclient: shutdown 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.187+0000 7f7a85fb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.187+0000 7f7a85fb9640 1 -- 172.21.15.117:0/3532190945 >> v1:172.21.15.117:6789/0 conn(0x7f7a8013f650 legacy=0x7f7a800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.187+0000 7f7a85fb9640 1 -- 172.21.15.117:0/3532190945 shutdown_connections 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.187+0000 7f7a85fb9640 1 -- 172.21.15.117:0/3532190945 >> 172.21.15.117:0/3532190945 conn(0x7f7a80066a00 msgr2=0x7f7a80066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.187+0000 7f7a85fb9640 1 -- 172.21.15.117:0/3532190945 shutdown_connections 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.187+0000 7f7a85fb9640 1 -- 172.21.15.117:0/3532190945 wait complete. 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.187+0000 7f7a85fb9640 10 monclient: build_initial_monmap 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.187+0000 7f7a85fb9640 10 monclient: monmap: 2024-03-29T10:34:18.183 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 1 librados: starting msgr at 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 1 librados: starting objecter 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 1 Processor -- start 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 1 -- start start 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 1 librados: setting wanted keys 2024-03-29T10:34:18.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 1 librados: calling monclient init 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 10 monclient: init 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.188+0000 7f7a85fb9640 10 monclient(hunting): picked mon.noname-b con 0x7f7a800a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 10 monclient(hunting): picked mon.noname-c con 0x7f7a8013f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 10 monclient(hunting): picked mon.noname-a con 0x7f7a801457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a800ad1b0 con 0x7f7a801457f0 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a80142da0 con 0x7f7a800a9d70 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8013fda0 con 0x7f7a8013f650 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 10 monclient(hunting): authenticate will time out at 779.451172s 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a7ffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7a801457f0 0x7f7a801e28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:51592/0 (socket says 172.21.15.117:51592) 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a7effd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f7a8013f650 0x7f7a80137f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58584/0 (socket says 172.21.15.117:58584) 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a7ffff640 1 -- 172.21.15.117:0/1393417121 learned_addr learned my addr 172.21.15.117:0/1393417121 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:18.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 client.?.objecter ms_handle_connect 0x7f7a801457f0 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 client.?.objecter ms_handle_connect 0x7f7a8013f650 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 client.?.objecter ms_handle_connect 0x7f7a800a9d70 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840656487 0 0) 0x7f7a800ad1b0 con 0x7f7a801457f0 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): my global_id is 4197 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6c004eb0 con 0x7f7a801457f0 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144151421 0 0) 0x7f7a8013fda0 con 0x7f7a8013f650 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:18.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): my global_id is 4199 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a800ad1b0 con 0x7f7a8013f650 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656261446 0 0) 0x7f7a80142da0 con 0x7f7a800a9d70 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): my global_id is 4192 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8013fda0 con 0x7f7a800a9d70 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4043441739 0 0) 0x7f7a6c004eb0 con 0x7f7a801457f0 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a80142da0 con 0x7f7a801457f0 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7a780016b0 con 0x7f7a801457f0 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:34:18.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): dump: 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4192638921 0 0) 0x7f7a800ad1b0 con 0x7f7a8013f650 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a6c004eb0 con 0x7f7a8013f650 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2021838806 0 0) 0x7f7a8013fda0 con 0x7f7a800a9d70 2024-03-29T10:34:18.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a800ad1b0 con 0x7f7a800a9d70 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7a74001ae0 con 0x7f7a8013f650 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): dump: 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:18.189 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7a700017d0 con 0x7f7a800a9d70 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient(hunting): dump: 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:18.190 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3296720649 0 0) 0x7f7a80142da0 con 0x7f7a801457f0 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 >> v1:172.21.15.117:6790/0 conn(0x7f7a8013f650 legacy=0x7f7a80137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 >> v1:172.21.15.136:6789/0 conn(0x7f7a800a9d70 legacy=0x7f7a801d7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient: _finish_hunting 0 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 monclient: found mon.a 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a801d7b20 con 0x7f7a801457f0 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient: _finish_auth 0 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 10 monclient: _check_auth_tickets 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 5 monclient: authenticate success, global_id 4197 2024-03-29T10:34:18.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 10 monclient: _renew_subs 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a801d7dd0 con 0x7f7a801457f0 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 10 monclient: _renew_subs 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a801429b0 con 0x7f7a801457f0 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.189+0000 7f7a85fb9640 1 librados: init done 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a78004b60 con 0x7f7a801457f0 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7a78004e70 con 0x7f7a801457f0 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a7dffb640 4 set_mon_vals no callback set 2024-03-29T10:34:18.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 10 monclient: dump: 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7a780053a0 con 0x7f7a801457f0 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f7a78006d00 con 0x7f7a801457f0 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 10 client.4197.objecter ms_dispatch 0x7f7a80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:34:18.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:34:18.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:34:18.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.190+0000 7f7a67fff640 20 client.4197.objecter dump_active .. 0 homeless 2024-03-29T10:34:18.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.191+0000 7f7a85fb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:18.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.191+0000 7f7a85fb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:18.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.191+0000 7f7a85fb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:18.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.191+0000 7f7a85fb9640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a60005080 con 0x7f7a801457f0 2024-03-29T10:34:18.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.193+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7a78006740 con 0x7f7a801457f0 2024-03-29T10:34:18.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.193+0000 7f7a67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:18.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.193+0000 7f7a67fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:18.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.255+0000 7f7a85fb9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:18.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.255+0000 7f7a85fb9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:18.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.255+0000 7f7a85fb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:18.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.255+0000 7f7a85fb9640 1 -- 172.21.15.117:0/1393417121 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a60005330 con 0x7f7a801457f0 2024-03-29T10:34:18.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.256+0000 7f7a67fff640 1 -- 172.21.15.117:0/1393417121 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f7a78006990 con 0x7f7a801457f0 2024-03-29T10:34:18.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.256+0000 7f7a67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:18.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.256+0000 7f7a67fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:18.242 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:34:18.242 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:34:18.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 10 librados: watch_flush enter 2024-03-29T10:34:18.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 10 librados: watch_flush exit 2024-03-29T10:34:18.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 20 client.4197.objecter shutdown clearing up homeless session... 2024-03-29T10:34:18.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 10 client.4197.objecter successfully canceled tick 2024-03-29T10:34:18.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 10 monclient: shutdown 2024-03-29T10:34:18.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:18.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 1 -- 172.21.15.117:0/1393417121 >> v1:172.21.15.117:6789/0 conn(0x7f7a801457f0 legacy=0x7f7a801e28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:18.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 1 -- 172.21.15.117:0/1393417121 shutdown_connections 2024-03-29T10:34:18.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 1 -- 172.21.15.117:0/1393417121 >> 172.21.15.117:0/1393417121 conn(0x7f7a80066a00 msgr2=0x7f7a80066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:18.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 1 -- 172.21.15.117:0/1393417121 shutdown_connections 2024-03-29T10:34:18.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 1 -- 172.21.15.117:0/1393417121 wait complete. 2024-03-29T10:34:18.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:18.257+0000 7f7a85fb9640 1 librados: shutdown 2024-03-29T10:34:18.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:34:18.590 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:18.605+0000 7f414a1ed040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:34:18.640 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:18.655+0000 7f2e34034040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:34:18.861 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:18.875+0000 7f414a1ed040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:34:18.913 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:18.928+0000 7f414a1ed040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:34:18.915 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:18.930+0000 7f2e34034040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:34:18.966 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:18.981+0000 7f2e34034040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:34:19.027 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:19.042+0000 7f414a1ed040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:34:19.085 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-29T10:34:19.100+0000 7f2e34034040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:34:20.041 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:20.055+0000 7f40eb63d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:34:20.043 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T10:34:20.058+0000 7f40eb63d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:34:21.131 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-03-29T10:34:21.146+0000 7f62c2060640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:34:21.185 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-03-29T10:34:21.199+0000 7fbfec61b640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:34:21.185 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-03-29T10:34:21.200+0000 7f1f3521e640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:34:21.195 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-03-29T10:34:21.210+0000 7f317d16f640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:34:21.197 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-03-29T10:34:21.212+0000 7f1eaf721640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:34:21.202 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-03-29T10:34:21.217+0000 7f9afab26640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:34:21.204 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-03-29T10:34:21.219+0000 7fcea500d640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:34:21.204 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-03-29T10:34:21.219+0000 7f424ff26640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:34:25.057 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:34:25.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.227+0000 7f327476c640 10 monclient: get_monmap_and_config 2024-03-29T10:34:25.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.227+0000 7f327476c640 10 monclient: build_initial_monmap 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.227+0000 7f327476c640 10 monclient: monmap: 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:34:25.228119+0000 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:34:25.228119+0000 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:25.213 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:25.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.228+0000 7f327476c640 1 Processor -- start 2024-03-29T10:34:25.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.228+0000 7f327476c640 1 -- start start 2024-03-29T10:34:25.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.228+0000 7f327476c640 10 monclient: init 2024-03-29T10:34:25.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:25.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:34:25.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 10 monclient(hunting): picked mon.noname-a con 0x7f326c070120 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:25.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 10 monclient(hunting): picked mon.noname-b con 0x7f326c0b0420 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 10 monclient(hunting): picked mon.noname-c con 0x7f326c070500 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c072090 con 0x7f326c070120 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c072370 con 0x7f326c070500 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c072650 con 0x7f326c0b0420 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f327476c640 10 monclient(hunting): authenticate will time out at 786.491428s 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f3271d07640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f326c0b0420 0x7f326c1405b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:39826/0 (socket says 172.21.15.117:39826) 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.229+0000 7f3271d07640 1 -- 172.21.15.117:0/3665634608 learned_addr learned my addr 172.21.15.117:0/3665634608 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629089004 0 0) 0x7f326c072650 con 0x7f326c0b0420 2024-03-29T10:34:25.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): _negotiate 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3258003520 con 0x7f326c0b0420 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724111674 0 0) 0x7f326c072370 con 0x7f326c070500 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): _negotiate 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): my global_id is 4231 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f326c072650 con 0x7f326c070500 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349111327 0 0) 0x7f326c072090 con 0x7f326c070120 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): _negotiate 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:25.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:25.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:34:25.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.230+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f326c072370 con 0x7f326c070120 2024-03-29T10:34:25.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 193892973 0 0) 0x7f326c072650 con 0x7f326c070500 2024-03-29T10:34:25.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 >> v1:172.21.15.117:6790/0 conn(0x7f326c0b0420 legacy=0x7f326c1405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:25.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 >> v1:172.21.15.117:6789/0 conn(0x7f326c070120 legacy=0x7f326c13d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:25.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 10 monclient: _finish_hunting 0 2024-03-29T10:34:25.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 1 monclient: found mon.noname-c 2024-03-29T10:34:25.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:25.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c140090 con 0x7f326c070500 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 10 monclient: _finish_auth 0 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 10 monclient: _check_auth_tickets 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f327476c640 5 monclient: authenticate success, global_id 4231 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f327476c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f325c001670 con 0x7f326c070500 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f3271506640 10 monclient: dump: 2024-03-29T10:34:25.224 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.231+0000 7f327476c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.232+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f325c001890 con 0x7f326c070500 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.232+0000 7f3271506640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.232+0000 7f327476c640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.232+0000 7f327476c640 4 set_mon_vals no callback set 2024-03-29T10:34:25.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.232+0000 7f3271506640 1 -- 172.21.15.117:0/3665634608 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f325c001ba0 con 0x7f326c070500 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.232+0000 7f3271506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.232+0000 7f3271506640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.232+0000 7f3271506640 10 monclient: dump: 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.233+0000 7f327476c640 10 monclient: shutdown 2024-03-29T10:34:25.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.233+0000 7f327476c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:25.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.233+0000 7f327476c640 1 -- 172.21.15.117:0/3665634608 >> v1:172.21.15.136:6789/0 conn(0x7f326c070500 legacy=0x7f326c14b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:25.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.233+0000 7f327476c640 1 -- 172.21.15.117:0/3665634608 shutdown_connections 2024-03-29T10:34:25.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.233+0000 7f327476c640 1 -- 172.21.15.117:0/3665634608 >> 172.21.15.117:0/3665634608 conn(0x7f326c067500 msgr2=0x7f326c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:25.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.233+0000 7f327476c640 1 -- 172.21.15.117:0/3665634608 shutdown_connections 2024-03-29T10:34:25.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.233+0000 7f327476c640 1 -- 172.21.15.117:0/3665634608 wait complete. 2024-03-29T10:34:25.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.234+0000 7f327476c640 10 monclient: build_initial_monmap 2024-03-29T10:34:25.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.234+0000 7f327476c640 10 monclient: monmap: 2024-03-29T10:34:25.235 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:25.235 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:25.235 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.235+0000 7f327476c640 1 librados: starting msgr at 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.235+0000 7f327476c640 1 librados: starting objecter 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.235+0000 7f327476c640 1 Processor -- start 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.235+0000 7f327476c640 1 -- start start 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.235+0000 7f327476c640 1 librados: setting wanted keys 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.235+0000 7f327476c640 1 librados: calling monclient init 2024-03-29T10:34:25.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.235+0000 7f327476c640 10 monclient: init 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 10 monclient(hunting): picked mon.noname-b con 0x7f326c070120 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 10 monclient(hunting): picked mon.noname-c con 0x7f326c070500 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 10 monclient(hunting): picked mon.noname-a con 0x7f326c0b0420 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c0b36d0 con 0x7f326c0b0420 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c0aba40 con 0x7f326c070120 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c0abd20 con 0x7f326c070500 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f3272508640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f326c070120 0x7f326c0aa800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:36754/0 (socket says 172.21.15.117:36754) 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f3272508640 1 -- 172.21.15.117:0/3110908837 learned_addr learned my addr 172.21.15.117:0/3110908837 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.236+0000 7f327476c640 10 monclient(hunting): authenticate will time out at 786.498812s 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 client.?.objecter ms_handle_connect 0x7f326c070120 2024-03-29T10:34:25.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 client.?.objecter ms_handle_connect 0x7f326c070500 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 client.?.objecter ms_handle_connect 0x7f326c0b0420 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040007278 0 0) 0x7f326c0abd20 con 0x7f326c070500 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.237+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3250004eb0 con 0x7f326c070500 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283804443 0 0) 0x7f326c0aba40 con 0x7f326c070120 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:25.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f326c0abd20 con 0x7f326c070120 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747756904 0 0) 0x7f326c0b36d0 con 0x7f326c0b0420 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f326c0aba40 con 0x7f326c0b0420 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1896871489 0 0) 0x7f3250004eb0 con 0x7f326c070500 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f326c0b36d0 con 0x7f326c070500 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3264001a90 con 0x7f326c070500 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): dump: 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:25.239 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 890479657 0 0) 0x7f326c0abd20 con 0x7f326c070120 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3250004eb0 con 0x7f326c070120 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3268001940 con 0x7f326c070120 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:25.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.238+0000 7f3262ffd640 10 monclient(hunting): dump: 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.239+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1975395887 0 0) 0x7f326c0aba40 con 0x7f326c0b0420 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.239+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f326c0abd20 con 0x7f326c0b0420 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.239+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f325c001530 con 0x7f326c0b0420 2024-03-29T10:34:25.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.239+0000 7f3262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.239+0000 7f3262ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.239+0000 7f3262ffd640 10 monclient(hunting): dump: 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.239+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 142285442 0 0) 0x7f3250004eb0 con 0x7f326c070120 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 >> v1:172.21.15.117:6790/0 conn(0x7f326c070500 legacy=0x7f326c0b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:25.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 >> v1:172.21.15.117:6789/0 conn(0x7f326c0b0420 legacy=0x7f326c0b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 1 monclient: found mon.b 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c0a9c10 con 0x7f326c070120 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 10 monclient: _finish_auth 0 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f3262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f327476c640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f327476c640 10 monclient: _renew_subs 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f327476c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f327476c640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326c0a9ec0 con 0x7f326c070120 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f327476c640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f327476c640 10 monclient: _renew_subs 2024-03-29T10:34:25.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f327476c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f327476c640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c1efbd0 con 0x7f326c070120 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.240+0000 7f327476c640 1 librados: init done 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.241+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32680035f0 con 0x7f326c070120 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.241+0000 7f3262ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.241+0000 7f3270d05640 4 set_mon_vals no callback set 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.241+0000 7f3260ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.242+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3268003980 con 0x7f326c070120 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.242+0000 7f3260ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.242+0000 7f3260ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.242+0000 7f3260ff9640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3254005080 con 0x7f326c070120 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.243+0000 7f3262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.243+0000 7f3262ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.243+0000 7f3262ffd640 10 monclient: dump: 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:25.244 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.243+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f3268003ae0 con 0x7f326c070120 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.244+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f3268084960 con 0x7f326c070120 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.244+0000 7f3262ffd640 10 client.4234.objecter ms_dispatch 0x7f326c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.244+0000 7f3262ffd640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.244+0000 7f3262ffd640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.244+0000 7f3262ffd640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T10:34:25.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.245+0000 7f3271d07640 10 monclient: get_auth_request con 0x7f325006e0b0 auth_method 0 2024-03-29T10:34:25.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.245+0000 7f3262ffd640 10 client.4234.objecter ms_handle_connect 0x7f325006e0b0 2024-03-29T10:34:25.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.251+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3268050240 con 0x7f326c070120 2024-03-29T10:34:25.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.251+0000 7f3262ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:25.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.251+0000 7f3262ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:25.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.615+0000 7f3260ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:25.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.616+0000 7f3260ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:25.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.616+0000 7f3260ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:34:25.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.616+0000 7f3260ff9640 1 -- 172.21.15.117:0/3110908837 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3254005330 con 0x7f326c070120 2024-03-29T10:34:25.602 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.617+0000 7f3262ffd640 1 -- 172.21.15.117:0/3110908837 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f3268052170 con 0x7f326c070120 2024-03-29T10:34:25.602 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.617+0000 7f3262ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:25.602 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.617+0000 7f3262ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:25.602 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:34:25.602 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:34:25.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.620+0000 7f3260ff9640 10 librados: watch_flush enter 2024-03-29T10:34:25.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 10 librados: watch_flush exit 2024-03-29T10:34:25.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T10:34:25.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 10 client.4234.objecter successfully canceled tick 2024-03-29T10:34:25.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 1 -- 172.21.15.117:0/3110908837 >> v1:172.21.15.117:6816/376768171 conn(0x7f325006e0b0 legacy=0x7f3250070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:25.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 10 monclient: shutdown 2024-03-29T10:34:25.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:25.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 1 -- 172.21.15.117:0/3110908837 >> v1:172.21.15.136:6789/0 conn(0x7f326c070120 legacy=0x7f326c0aa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:25.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 1 -- 172.21.15.117:0/3110908837 shutdown_connections 2024-03-29T10:34:25.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 1 -- 172.21.15.117:0/3110908837 >> 172.21.15.117:0/3110908837 conn(0x7f326c067500 msgr2=0x7f326c13c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:25.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.621+0000 7f3260ff9640 1 -- 172.21.15.117:0/3110908837 shutdown_connections 2024-03-29T10:34:25.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.622+0000 7f3260ff9640 1 -- 172.21.15.117:0/3110908837 wait complete. 2024-03-29T10:34:25.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:25.623+0000 7f3260ff9640 1 librados: shutdown 2024-03-29T10:34:25.624 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:34:32.432 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:34:32.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.577+0000 7f8fa6af1640 10 monclient: get_monmap_and_config 2024-03-29T10:34:32.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.577+0000 7f8fa6af1640 10 monclient: build_initial_monmap 2024-03-29T10:34:32.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.577+0000 7f8fa6af1640 10 monclient: monmap: 2024-03-29T10:34:32.567 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:32.567 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:32.567 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:34:32.578637+0000 2024-03-29T10:34:32.567 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:34:32.578637+0000 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.579+0000 7f8fa6af1640 1 Processor -- start 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.579+0000 7f8fa6af1640 1 -- start start 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.579+0000 7f8fa6af1640 10 monclient: init 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa01422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa0142690 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa00b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:32.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa0142c80 con 0x7f8fa00b17d0 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00aae40 con 0x7f8fa01422b0 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa013fbd0 con 0x7f8fa0142690 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa6af1640 10 monclient(hunting): authenticate will time out at 793.842229s 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f8fa0142690 0x7f8fa00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:56948/0 (socket says 172.21.15.117:56948) 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9ffff640 1 -- 172.21.15.117:0/1301513372 learned_addr learned my addr 172.21.15.117:0/1301513372 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8fa488d640 1 --1- 172.21.15.117:0/1301513372 >> v1:172.21.15.136:6789/0 conn(0x7f8fa01422b0 0x7f8fa013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:34962/0 (socket says 172.21.15.117:34962) 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940596820 0 0) 0x7f8fa013fbd0 con 0x7f8fa0142690 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f90003520 con 0x7f8fa0142690 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196444252 0 0) 0x7f8fa00aae40 con 0x7f8fa01422b0 2024-03-29T10:34:32.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:34:32.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:32.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:32.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:34:32.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa013fbd0 con 0x7f8fa01422b0 2024-03-29T10:34:32.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3782026400 0 0) 0x7f8f90003520 con 0x7f8fa0142690 2024-03-29T10:34:32.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 >> v1:172.21.15.136:6789/0 conn(0x7f8fa01422b0 legacy=0x7f8fa013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:32.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.580+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 >> v1:172.21.15.117:6789/0 conn(0x7f8fa00b17d0 legacy=0x7f8fa00b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:34:32.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:34:32.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:34:32.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:32.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa0151940 con 0x7f8fa0142690 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8f8c001690 con 0x7f8fa0142690 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8fa6af1640 5 monclient: authenticate success, global_id 4247 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: dump: 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:32.573 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8fa6af1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f8c0017f0 con 0x7f8fa0142690 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 1 -- 172.21.15.117:0/1301513372 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8f8c001b80 con 0x7f8fa0142690 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8fa6af1640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8fa6af1640 4 set_mon_vals no callback set 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.583+0000 7f8f9f7fe640 10 monclient: dump: 2024-03-29T10:34:32.574 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.584+0000 7f8fa6af1640 10 monclient: shutdown 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.584+0000 7f8fa6af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.584+0000 7f8fa6af1640 1 -- 172.21.15.117:0/1301513372 >> v1:172.21.15.117:6790/0 conn(0x7f8fa0142690 legacy=0x7f8fa00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.584+0000 7f8fa6af1640 1 -- 172.21.15.117:0/1301513372 shutdown_connections 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.585+0000 7f8fa6af1640 1 -- 172.21.15.117:0/1301513372 >> 172.21.15.117:0/1301513372 conn(0x7f8fa0066a00 msgr2=0x7f8fa0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:32.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.585+0000 7f8fa6af1640 1 -- 172.21.15.117:0/1301513372 shutdown_connections 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.585+0000 7f8fa6af1640 1 -- 172.21.15.117:0/1301513372 wait complete. 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.585+0000 7f8fa6af1640 10 monclient: build_initial_monmap 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.585+0000 7f8fa6af1640 10 monclient: monmap: 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.586+0000 7f8fa6af1640 1 librados: starting msgr at 2024-03-29T10:34:32.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.586+0000 7f8fa6af1640 1 librados: starting objecter 2024-03-29T10:34:32.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.586+0000 7f8fa6af1640 1 Processor -- start 2024-03-29T10:34:32.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.586+0000 7f8fa6af1640 1 -- start start 2024-03-29T10:34:32.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.586+0000 7f8fa6af1640 1 librados: setting wanted keys 2024-03-29T10:34:32.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.586+0000 7f8fa6af1640 1 librados: calling monclient init 2024-03-29T10:34:32.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.586+0000 7f8fa6af1640 10 monclient: init 2024-03-29T10:34:32.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8fa6af1640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:32.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8fa6af1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:32.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8fa6af1640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa00b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:32.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8fa488d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f8fa00b17d0 0x7f8fa00ab7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:57788/0 (socket says 172.21.15.117:57788) 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8fa6af1640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa01422b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8fa488d640 1 -- 172.21.15.117:0/3273598675 learned_addr learned my addr 172.21.15.117:0/3273598675 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8fa6af1640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa0142690 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00ac730 con 0x7f8fa01422b0 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00ac980 con 0x7f8fa0142690 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.587+0000 7f8f9d7fa640 10 client.?.objecter ms_handle_connect 0x7f8fa00b17d0 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8f9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00acc60 con 0x7f8fa00b17d0 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8fa6af1640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8fa6af1640 10 monclient(hunting): authenticate will time out at 793.850249s 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992416897 0 0) 0x7f8fa00acc60 con 0x7f8fa00b17d0 2024-03-29T10:34:32.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8f9d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8f9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8f9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8f9d7fa640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.588+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f84004eb0 con 0x7f8fa00b17d0 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.589+0000 7f8f9d7fa640 10 client.?.objecter ms_handle_connect 0x7f8fa0142690 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.589+0000 7f8f9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.589+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1525981663 0 0) 0x7f8f84004eb0 con 0x7f8fa00b17d0 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.589+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fa00acc60 con 0x7f8fa00b17d0 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.589+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8f940019f0 con 0x7f8fa00b17d0 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.589+0000 7f8f9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.589+0000 7f8f9d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.589+0000 7f8f9d7fa640 10 monclient(hunting): dump: 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:32.582 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.589+0000 7f8f9d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.590+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2897238388 0 0) 0x7f8fa00acc60 con 0x7f8fa00b17d0 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.590+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 >> v1:172.21.15.136:6789/0 conn(0x7f8fa0142690 legacy=0x7f8fa012d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:32.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.590+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 >> v1:172.21.15.117:6789/0 conn(0x7f8fa01422b0 legacy=0x7f8fa012e0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:34:32.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8f9d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:34:32.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8f9d7fa640 1 monclient: found mon.c 2024-03-29T10:34:32.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8f9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:32.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8f9d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:32.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00acf40 con 0x7f8fa00b17d0 2024-03-29T10:34:32.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8f9d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:34:32.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8f9d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:34:32.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8f9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:32.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8fa6af1640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:34:32.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8fa6af1640 10 monclient: _renew_subs 2024-03-29T10:34:32.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8fa6af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:32.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa01323d0 con 0x7f8fa00b17d0 2024-03-29T10:34:32.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8fa6af1640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:32.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8fa6af1640 10 monclient: _renew_subs 2024-03-29T10:34:32.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8fa6af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:32.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa0151090 con 0x7f8fa00b17d0 2024-03-29T10:34:32.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.591+0000 7f8fa6af1640 1 librados: init done 2024-03-29T10:34:32.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.592+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f94001cb0 con 0x7f8fa00b17d0 2024-03-29T10:34:32.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.592+0000 7f8f9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:32.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.592+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8f94004c30 con 0x7f8fa00b17d0 2024-03-29T10:34:32.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.592+0000 7f8f9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:32.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.592+0000 7f8f9d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:32.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.592+0000 7f8f9effd640 4 set_mon_vals no callback set 2024-03-29T10:34:32.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.592+0000 7f8f9d7fa640 10 monclient: dump: 2024-03-29T10:34:32.587 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:32.588 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:32.588 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:32.588 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:32.588 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:32.588 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:32.588 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:32.589 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:32.589 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:32.589 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:32.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.593+0000 7f8fa6af1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:32.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.593+0000 7f8fa6af1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:32.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.593+0000 7f8fa6af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:32.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.593+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f80005080 con 0x7f8fa00b17d0 2024-03-29T10:34:32.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.594+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f8f94004d90 con 0x7f8fa00b17d0 2024-03-29T10:34:32.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.595+0000 7f8f9ffff640 10 monclient: get_auth_request con 0x7f8f8406dcb0 auth_method 0 2024-03-29T10:34:32.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.595+0000 7f8f9d7fa640 10 client.4250.objecter ms_handle_connect 0x7f8f8406dcb0 2024-03-29T10:34:32.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.595+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f8f94084bb0 con 0x7f8fa00b17d0 2024-03-29T10:34:32.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.595+0000 7f8f9d7fa640 10 client.4250.objecter ms_dispatch 0x7f8fa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:34:32.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.595+0000 7f8f9d7fa640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:34:32.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.595+0000 7f8f9d7fa640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:34:32.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.595+0000 7f8f9d7fa640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T10:34:32.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.601+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f94050490 con 0x7f8fa00b17d0 2024-03-29T10:34:32.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.601+0000 7f8f9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:32.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.601+0000 7f8f9d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:32.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.903+0000 7f8fa6af1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:32.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.903+0000 7f8fa6af1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:32.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.903+0000 7f8fa6af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:34:32.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.903+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f80005330 con 0x7f8fa00b17d0 2024-03-29T10:34:32.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.903+0000 7f8f9d7fa640 1 -- 172.21.15.117:0/3273598675 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f8f940523c0 con 0x7f8fa00b17d0 2024-03-29T10:34:32.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.903+0000 7f8f9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:32.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.903+0000 7f8f9d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:32.889 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:34:32.889 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.906+0000 7f8fa6af1640 10 librados: watch_flush enter 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.906+0000 7f8fa6af1640 10 librados: watch_flush exit 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.906+0000 7f8fa6af1640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.906+0000 7f8fa6af1640 10 client.4250.objecter successfully canceled tick 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.907+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 >> v1:172.21.15.117:6816/376768171 conn(0x7f8f8406dcb0 legacy=0x7f8f84070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.907+0000 7f8fa6af1640 10 monclient: shutdown 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.907+0000 7f8fa6af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.907+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 >> v1:172.21.15.117:6790/0 conn(0x7f8fa00b17d0 legacy=0x7f8fa00ab7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.907+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 shutdown_connections 2024-03-29T10:34:32.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.907+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 >> 172.21.15.117:0/3273598675 conn(0x7f8fa0066a00 msgr2=0x7f8fa013dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:32.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.907+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 shutdown_connections 2024-03-29T10:34:32.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.907+0000 7f8fa6af1640 1 -- 172.21.15.117:0/3273598675 wait complete. 2024-03-29T10:34:32.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:32.907+0000 7f8fa6af1640 1 librados: shutdown 2024-03-29T10:34:32.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:34:39.714 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.880+0000 7f4aae372640 10 monclient: get_monmap_and_config 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.880+0000 7f4aae372640 10 monclient: build_initial_monmap 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.880+0000 7f4aae372640 10 monclient: monmap: 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:34:39.881645+0000 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:34:39.881645+0000 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:39.866 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:39.867 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:39.867 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:39.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.881+0000 7f4aae372640 1 Processor -- start 2024-03-29T10:34:39.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.881+0000 7f4aae372640 1 -- start start 2024-03-29T10:34:39.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.881+0000 7f4aae372640 10 monclient: init 2024-03-29T10:34:39.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa80b4330 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa81470b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa81439b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa806f2a0 con 0x7f4aa81439b0 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa8146b90 con 0x7f4aa80b4330 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa80b2ca0 con 0x7f4aa81470b0 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.882+0000 7f4aae372640 10 monclient(hunting): authenticate will time out at 801.144690s 2024-03-29T10:34:39.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.883+0000 7f4aa7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4aa80b4330 0x7f4aa80b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:46410/0 (socket says 172.21.15.117:46410) 2024-03-29T10:34:39.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.883+0000 7f4aa7fff640 1 -- 172.21.15.117:0/2270194274 learned_addr learned my addr 172.21.15.117:0/2270194274 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:39.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.885+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930495458 0 0) 0x7f4aa8146b90 con 0x7f4aa80b4330 2024-03-29T10:34:39.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:39.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:39.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:39.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:34:39.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a90003520 con 0x7f4aa80b4330 2024-03-29T10:34:39.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305023107 0 0) 0x7f4aa80b2ca0 con 0x7f4aa81470b0 2024-03-29T10:34:39.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:39.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:39.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:39.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:34:39.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa8146b90 con 0x7f4aa81470b0 2024-03-29T10:34:39.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2095950129 0 0) 0x7f4a90003520 con 0x7f4aa80b4330 2024-03-29T10:34:39.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 >> v1:172.21.15.117:6790/0 conn(0x7f4aa81470b0 legacy=0x7f4aa80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:39.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.886+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 >> v1:172.21.15.117:6789/0 conn(0x7f4aa81439b0 legacy=0x7f4aa8151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:34:39.873 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:39.873 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 1 monclient: found mon.noname-c 2024-03-29T10:34:39.873 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:39.873 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:39.873 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa806e0c0 con 0x7f4aa80b4330 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aae372640 5 monclient: authenticate success, global_id 4243 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aae372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4a9c001610 con 0x7f4aa80b4330 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aa6ffd640 10 monclient: dump: 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:39.874 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:39.875 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:39.875 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:39.875 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:39.875 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:39.875 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:39.875 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:39.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.888+0000 7f4aae372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:39.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.889+0000 7f4aa6ffd640 1 -- 172.21.15.117:0/2270194274 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a9c001770 con 0x7f4aa80b4330 2024-03-29T10:34:39.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.889+0000 7f4aa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:39.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.889+0000 7f4aae372640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:39.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.889+0000 7f4aae372640 4 set_mon_vals no callback set 2024-03-29T10:34:39.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.890+0000 7f4aae372640 10 monclient: shutdown 2024-03-29T10:34:39.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.890+0000 7f4aae372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:39.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.890+0000 7f4aae372640 1 -- 172.21.15.117:0/2270194274 >> v1:172.21.15.136:6789/0 conn(0x7f4aa80b4330 legacy=0x7f4aa80b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:39.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.890+0000 7f4aae372640 1 -- 172.21.15.117:0/2270194274 shutdown_connections 2024-03-29T10:34:39.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.890+0000 7f4aae372640 1 -- 172.21.15.117:0/2270194274 >> 172.21.15.117:0/2270194274 conn(0x7f4aa8066a00 msgr2=0x7f4aa8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:39.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.891+0000 7f4aae372640 1 -- 172.21.15.117:0/2270194274 shutdown_connections 2024-03-29T10:34:39.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.891+0000 7f4aae372640 1 -- 172.21.15.117:0/2270194274 wait complete. 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.892+0000 7f4aae372640 10 monclient: build_initial_monmap 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.892+0000 7f4aae372640 10 monclient: monmap: 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:39.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.892+0000 7f4aae372640 1 librados: starting msgr at 2024-03-29T10:34:39.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.892+0000 7f4aae372640 1 librados: starting objecter 2024-03-29T10:34:39.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.893+0000 7f4aae372640 1 Processor -- start 2024-03-29T10:34:39.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.893+0000 7f4aae372640 1 -- start start 2024-03-29T10:34:39.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.893+0000 7f4aae372640 1 librados: setting wanted keys 2024-03-29T10:34:39.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.893+0000 7f4aae372640 1 librados: calling monclient init 2024-03-29T10:34:39.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.893+0000 7f4aae372640 10 monclient: init 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa80b4330 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa81439b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa81470b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa81df950 con 0x7f4aa81470b0 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa81dfc10 con 0x7f4aa80b4330 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa8127840 con 0x7f4aa81439b0 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:39.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.894+0000 7f4aae372640 10 monclient(hunting): authenticate will time out at 801.156498s 2024-03-29T10:34:39.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.895+0000 7f4aa7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4aa80b4330 0x7f4aa81df240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:46754/0 (socket says 172.21.15.117:46754) 2024-03-29T10:34:39.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.895+0000 7f4aa7fff640 1 -- 172.21.15.117:0/1624349291 learned_addr learned my addr 172.21.15.117:0/1624349291 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:39.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.896+0000 7f4aa4ff9640 10 client.?.objecter ms_handle_connect 0x7f4aa80b4330 2024-03-29T10:34:39.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.896+0000 7f4aa4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:39.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 10 client.?.objecter ms_handle_connect 0x7f4aa81470b0 2024-03-29T10:34:39.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:39.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 10 client.?.objecter ms_handle_connect 0x7f4aa81439b0 2024-03-29T10:34:39.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:39.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746090235 0 0) 0x7f4aa81df950 con 0x7f4aa81470b0 2024-03-29T10:34:39.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:39.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:39.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:39.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:34:39.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.897+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a94004eb0 con 0x7f4aa81470b0 2024-03-29T10:34:39.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.898+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372664993 0 0) 0x7f4aa81dfc10 con 0x7f4aa80b4330 2024-03-29T10:34:39.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.898+0000 7f4aa4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:39.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.898+0000 7f4aa4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:39.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.898+0000 7f4aa4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:39.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.898+0000 7f4aa4ff9640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:34:39.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.898+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa81df950 con 0x7f4aa80b4330 2024-03-29T10:34:39.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2846641085 0 0) 0x7f4a94004eb0 con 0x7f4aa81470b0 2024-03-29T10:34:39.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4aa81dfc10 con 0x7f4aa81470b0 2024-03-29T10:34:39.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643319161 0 0) 0x7f4aa8127840 con 0x7f4aa81439b0 2024-03-29T10:34:39.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a94004eb0 con 0x7f4aa81439b0 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4aa0001ac0 con 0x7f4aa81470b0 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 10 monclient(hunting): dump: 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:39.885 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:39.886 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:39.886 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:39.886 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:39.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.899+0000 7f4aa4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:39.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069950267 0 0) 0x7f4aa81df950 con 0x7f4aa80b4330 2024-03-29T10:34:39.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4aa8127840 con 0x7f4aa80b4330 2024-03-29T10:34:39.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1575869906 0 0) 0x7f4a94004eb0 con 0x7f4aa81439b0 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4aa81df950 con 0x7f4aa81439b0 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2787758668 0 0) 0x7f4aa81dfc10 con 0x7f4aa81470b0 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 >> v1:172.21.15.117:6790/0 conn(0x7f4aa81439b0 legacy=0x7f4aa81402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 >> v1:172.21.15.136:6789/0 conn(0x7f4aa80b4330 legacy=0x7f4aa81df240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 1 monclient: found mon.a 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa8140fb0 con 0x7f4aa81470b0 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aa4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aae372640 5 monclient: authenticate success, global_id 4248 2024-03-29T10:34:39.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aae372640 10 monclient: _renew_subs 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aae372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa8141280 con 0x7f4aa81470b0 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aae372640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aae372640 10 monclient: _renew_subs 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aae372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa8140b90 con 0x7f4aa81470b0 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.900+0000 7f4aae372640 1 librados: init done 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.901+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aa0001fd0 con 0x7f4aa81470b0 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.901+0000 7f4aa4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.901+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4aa0003b30 con 0x7f4aa81470b0 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.901+0000 7f4aa4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.901+0000 7f4aa4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:39.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.901+0000 7f4aa4ff9640 10 monclient: dump: 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:39.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.901+0000 7f4aa67fc640 4 set_mon_vals no callback set 2024-03-29T10:34:39.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aae372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f4aa0019b00 con 0x7f4aa81470b0 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aae372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aae372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a88005080 con 0x7f4aa81470b0 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f4aa0083990 con 0x7f4aa81470b0 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aa4ff9640 10 client.4248.objecter ms_dispatch 0x7f4aa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aa4ff9640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aa4ff9640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.903+0000 7f4aa4ff9640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T10:34:39.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.905+0000 7f4aa7fff640 10 monclient: get_auth_request con 0x7f4a9406deb0 auth_method 0 2024-03-29T10:34:39.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.906+0000 7f4aa4ff9640 10 client.4248.objecter ms_handle_connect 0x7f4a9406deb0 2024-03-29T10:34:39.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.911+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4aa004f270 con 0x7f4aa81470b0 2024-03-29T10:34:39.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.912+0000 7f4aa4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:39.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:39.912+0000 7f4aa4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:40.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.231+0000 7f4aae372640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:40.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.231+0000 7f4aae372640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:40.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.231+0000 7f4aae372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:40.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.231+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a88005330 con 0x7f4aa81470b0 2024-03-29T10:34:40.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.232+0000 7f4aa4ff9640 1 -- 172.21.15.117:0/1624349291 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f4aa00511a0 con 0x7f4aa81470b0 2024-03-29T10:34:40.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.232+0000 7f4aa4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:40.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.232+0000 7f4aa4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:40.218 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:34:40.218 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:34:40.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.235+0000 7f4aae372640 10 librados: watch_flush enter 2024-03-29T10:34:40.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.235+0000 7f4aae372640 10 librados: watch_flush exit 2024-03-29T10:34:40.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.235+0000 7f4aae372640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T10:34:40.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.235+0000 7f4aae372640 10 client.4248.objecter successfully canceled tick 2024-03-29T10:34:40.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.235+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 >> v1:172.21.15.117:6816/376768171 conn(0x7f4a9406deb0 legacy=0x7f4a94070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:40.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.235+0000 7f4aae372640 10 monclient: shutdown 2024-03-29T10:34:40.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.235+0000 7f4aae372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:40.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.235+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 >> v1:172.21.15.117:6789/0 conn(0x7f4aa81470b0 legacy=0x7f4aa8125250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:40.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.236+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 shutdown_connections 2024-03-29T10:34:40.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.236+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 >> 172.21.15.117:0/1624349291 conn(0x7f4aa8066a00 msgr2=0x7f4aa8070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:40.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.236+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 shutdown_connections 2024-03-29T10:34:40.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.236+0000 7f4aae372640 1 -- 172.21.15.117:0/1624349291 wait complete. 2024-03-29T10:34:40.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:40.236+0000 7f4aae372640 1 librados: shutdown 2024-03-29T10:34:40.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:34:47.047 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:34:47.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.210+0000 7f0135bef640 10 monclient: get_monmap_and_config 2024-03-29T10:34:47.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.210+0000 7f0135bef640 10 monclient: build_initial_monmap 2024-03-29T10:34:47.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.211+0000 7f0135bef640 10 monclient: monmap: 2024-03-29T10:34:47.196 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:47.196 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:47.196 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:34:47.211872+0000 2024-03-29T10:34:47.196 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:34:47.211872+0000 2024-03-29T10:34:47.197 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:47.197 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:47.197 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:47.197 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:47.197 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:47.197 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.212+0000 7f0135bef640 1 Processor -- start 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.212+0000 7f0135bef640 1 -- start start 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.212+0000 7f0135bef640 10 monclient: init 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.212+0000 7f0135bef640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.213+0000 7f0135bef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.213+0000 7f0135bef640 10 monclient(hunting): picked mon.noname-b con 0x7f01300b0860 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.213+0000 7f0135bef640 10 monclient(hunting): picked mon.noname-c con 0x7f013013ef30 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.213+0000 7f0135bef640 10 monclient(hunting): picked mon.noname-a con 0x7f01300af8e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.213+0000 7f0135bef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01300b0c40 con 0x7f01300af8e0 2024-03-29T10:34:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.213+0000 7f0135bef640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01300aed10 con 0x7f013013ef30 2024-03-29T10:34:47.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.213+0000 7f0135bef640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01300afd50 con 0x7f01300b0860 2024-03-29T10:34:47.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.213+0000 7f0135bef640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:47.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.213+0000 7f0135bef640 10 monclient(hunting): authenticate will time out at 808.475184s 2024-03-29T10:34:47.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.216+0000 7f012effd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f013013ef30 0x7f01300af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:58106/0 (socket says 172.21.15.117:58106) 2024-03-29T10:34:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.216+0000 7f012effd640 1 -- 172.21.15.117:0/1419648016 learned_addr learned my addr 172.21.15.117:0/1419648016 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.217+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963288396 0 0) 0x7f01300aed10 con 0x7f013013ef30 2024-03-29T10:34:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.217+0000 7f012e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.217+0000 7f012e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.217+0000 7f012e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.217+0000 7f012e7fc640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:34:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.217+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0118003520 con 0x7f013013ef30 2024-03-29T10:34:47.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170842341 0 0) 0x7f01300b0c40 con 0x7f01300af8e0 2024-03-29T10:34:47.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:47.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:47.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:47.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:34:47.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01300aed10 con 0x7f01300af8e0 2024-03-29T10:34:47.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1694343690 0 0) 0x7f0118003520 con 0x7f013013ef30 2024-03-29T10:34:47.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 >> v1:172.21.15.117:6790/0 conn(0x7f01300b0860 legacy=0x7f013013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:34:47.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 >> v1:172.21.15.117:6789/0 conn(0x7f01300af8e0 legacy=0x7f0130149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:47.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:47.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 1 monclient: found mon.noname-c 2024-03-29T10:34:47.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:47.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300ab1b0 con 0x7f013013ef30 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f0135bef640 5 monclient: authenticate success, global_id 4249 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0120001690 con 0x7f013013ef30 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f0135bef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f012e7fc640 10 monclient: dump: 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:47.206 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.218+0000 7f0135bef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.219+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01200018b0 con 0x7f013013ef30 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.219+0000 7f012e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.219+0000 7f0135bef640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.219+0000 7f0135bef640 4 set_mon_vals no callback set 2024-03-29T10:34:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.219+0000 7f012e7fc640 1 -- 172.21.15.117:0/1419648016 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0120001c40 con 0x7f013013ef30 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.219+0000 7f012e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.219+0000 7f012e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.219+0000 7f012e7fc640 10 monclient: dump: 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.220+0000 7f0135bef640 10 monclient: shutdown 2024-03-29T10:34:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.220+0000 7f0135bef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:47.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.220+0000 7f0135bef640 1 -- 172.21.15.117:0/1419648016 >> v1:172.21.15.136:6789/0 conn(0x7f013013ef30 legacy=0x7f01300af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:47.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.220+0000 7f0135bef640 1 -- 172.21.15.117:0/1419648016 shutdown_connections 2024-03-29T10:34:47.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.220+0000 7f0135bef640 1 -- 172.21.15.117:0/1419648016 >> 172.21.15.117:0/1419648016 conn(0x7f013006b660 msgr2=0x7f013006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:47.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.220+0000 7f0135bef640 1 -- 172.21.15.117:0/1419648016 shutdown_connections 2024-03-29T10:34:47.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.221+0000 7f0135bef640 1 -- 172.21.15.117:0/1419648016 wait complete. 2024-03-29T10:34:47.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.222+0000 7f0135bef640 10 monclient: build_initial_monmap 2024-03-29T10:34:47.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.222+0000 7f0135bef640 10 monclient: monmap: 2024-03-29T10:34:47.210 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:47.210 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:47.210 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:34:47.210 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.222+0000 7f0135bef640 1 librados: starting msgr at 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.222+0000 7f0135bef640 1 librados: starting objecter 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.222+0000 7f0135bef640 1 Processor -- start 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.222+0000 7f0135bef640 1 -- start start 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.222+0000 7f0135bef640 1 librados: setting wanted keys 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.222+0000 7f0135bef640 1 librados: calling monclient init 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.222+0000 7f0135bef640 10 monclient: init 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:47.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 10 monclient(hunting): picked mon.noname-b con 0x7f01300af8e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 10 monclient(hunting): picked mon.noname-c con 0x7f01300b0860 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 10 monclient(hunting): picked mon.noname-a con 0x7f013013ef30 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01301d74b0 con 0x7f013013ef30 2024-03-29T10:34:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01301d8330 con 0x7f01300af8e0 2024-03-29T10:34:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01301d7700 con 0x7f01300b0860 2024-03-29T10:34:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.223+0000 7f0135bef640 10 monclient(hunting): authenticate will time out at 808.485831s 2024-03-29T10:34:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.226+0000 7f012f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f01300af8e0 0x7f01301cac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:58218/0 (socket says 172.21.15.117:58218) 2024-03-29T10:34:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.226+0000 7f012f7fe640 1 -- 172.21.15.117:0/4245932660 learned_addr learned my addr 172.21.15.117:0/4245932660 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.226+0000 7f0117fff640 10 client.?.objecter ms_handle_connect 0x7f013013ef30 2024-03-29T10:34:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 client.?.objecter ms_handle_connect 0x7f01300b0860 2024-03-29T10:34:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665509486 0 0) 0x7f01301d74b0 con 0x7f013013ef30 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f011c004eb0 con 0x7f013013ef30 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878475635 0 0) 0x7f01301d7700 con 0x7f01300b0860 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f011c0051d0 con 0x7f01300b0860 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 194805032 0 0) 0x7f011c004eb0 con 0x7f013013ef30 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f011c005640 con 0x7f013013ef30 2024-03-29T10:34:47.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 client.?.objecter ms_handle_connect 0x7f01300af8e0 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0128001cb0 con 0x7f013013ef30 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): dump: 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:47.217 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.227+0000 7f0117fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 810053041 0 0) 0x7f011c005640 con 0x7f013013ef30 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 >> v1:172.21.15.117:6790/0 conn(0x7f01300b0860 legacy=0x7f01301d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 >> v1:172.21.15.136:6789/0 conn(0x7f01300af8e0 legacy=0x7f01301cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 10 monclient: _finish_hunting 0 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 1 monclient: found mon.a 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01301c6670 con 0x7f013013ef30 2024-03-29T10:34:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 10 monclient: _finish_auth 0 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 10 monclient: _check_auth_tickets 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0117fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0135bef640 5 monclient: authenticate success, global_id 4254 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0135bef640 10 monclient: _renew_subs 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0135bef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01301c5e00 con 0x7f013013ef30 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0135bef640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0135bef640 10 monclient: _renew_subs 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0135bef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01301c70f0 con 0x7f013013ef30 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.228+0000 7f0135bef640 1 librados: init done 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.229+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0128003970 con 0x7f013013ef30 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.229+0000 7f0117fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.229+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0128003c80 con 0x7f013013ef30 2024-03-29T10:34:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.229+0000 7f0117fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.229+0000 7f0117fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.229+0000 7f0117fff640 10 monclient: dump: 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:47.220 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:47.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.230+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f0128019cd0 con 0x7f013013ef30 2024-03-29T10:34:47.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.231+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f0128083920 con 0x7f013013ef30 2024-03-29T10:34:47.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.231+0000 7f0117fff640 10 client.4254.objecter ms_dispatch 0x7f0130000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:34:47.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.231+0000 7f0117fff640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:34:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.231+0000 7f0117fff640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:34:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.231+0000 7f0117fff640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-29T10:34:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.231+0000 7f012f7fe640 10 monclient: get_auth_request con 0x7f011c06e2a0 auth_method 0 2024-03-29T10:34:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.231+0000 7f012dffb640 4 set_mon_vals no callback set 2024-03-29T10:34:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.232+0000 7f0117fff640 10 client.4254.objecter ms_handle_connect 0x7f011c06e2a0 2024-03-29T10:34:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.233+0000 7f0135bef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.233+0000 7f0135bef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.233+0000 7f0135bef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.233+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0110005080 con 0x7f013013ef30 2024-03-29T10:34:47.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.241+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f012804f340 con 0x7f013013ef30 2024-03-29T10:34:47.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.241+0000 7f0117fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:47.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.241+0000 7f0117fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:47.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.551+0000 7f0135bef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:47.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.551+0000 7f0135bef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:47.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.551+0000 7f0135bef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:47.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.551+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0110005330 con 0x7f013013ef30 2024-03-29T10:34:47.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.552+0000 7f0117fff640 1 -- 172.21.15.117:0/4245932660 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f0128003ae0 con 0x7f013013ef30 2024-03-29T10:34:47.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.552+0000 7f0117fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:47.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.552+0000 7f0117fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:47.538 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:34:47.538 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:34:47.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.556+0000 7f0135bef640 10 librados: watch_flush enter 2024-03-29T10:34:47.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.556+0000 7f0135bef640 10 librados: watch_flush exit 2024-03-29T10:34:47.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.556+0000 7f0135bef640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-29T10:34:47.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.556+0000 7f0135bef640 10 client.4254.objecter successfully canceled tick 2024-03-29T10:34:47.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.557+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 >> v1:172.21.15.117:6816/376768171 conn(0x7f011c06e2a0 legacy=0x7f011c070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:47.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.557+0000 7f0135bef640 10 monclient: shutdown 2024-03-29T10:34:47.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.557+0000 7f0135bef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:47.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.557+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 >> v1:172.21.15.117:6789/0 conn(0x7f013013ef30 legacy=0x7f01301c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:47.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.557+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 shutdown_connections 2024-03-29T10:34:47.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.557+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 >> 172.21.15.117:0/4245932660 conn(0x7f013006b660 msgr2=0x7f013013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:47.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.558+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 shutdown_connections 2024-03-29T10:34:47.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.558+0000 7f0135bef640 1 -- 172.21.15.117:0/4245932660 wait complete. 2024-03-29T10:34:47.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:47.558+0000 7f0135bef640 1 librados: shutdown 2024-03-29T10:34:47.562 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:34:54.369 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:34:54.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.528+0000 7f0d55b13640 10 monclient: get_monmap_and_config 2024-03-29T10:34:54.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.528+0000 7f0d55b13640 10 monclient: build_initial_monmap 2024-03-29T10:34:54.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.528+0000 7f0d55b13640 10 monclient: monmap: 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:34:54.530505+0000 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:34:54.530505+0000 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:34:54.516 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:54.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.529+0000 7f0d55b13640 1 Processor -- start 2024-03-29T10:34:54.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.529+0000 7f0d55b13640 1 -- start start 2024-03-29T10:34:54.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.529+0000 7f0d55b13640 10 monclient: init 2024-03-29T10:34:54.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.530+0000 7f0d55b13640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:54.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.530+0000 7f0d55b13640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:54.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.533+0000 7f0d55b13640 10 monclient(hunting): picked mon.noname-a con 0x7f0d500a9ac0 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:54.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.533+0000 7f0d55b13640 10 monclient(hunting): picked mon.noname-c con 0x7f0d5013ee90 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:54.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.535+0000 7f0d55b13640 10 monclient(hunting): picked mon.noname-b con 0x7f0d500af900 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:54.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.535+0000 7f0d55b13640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d500afd10 con 0x7f0d500a9ac0 2024-03-29T10:34:54.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.535+0000 7f0d55b13640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d500af6f0 con 0x7f0d5013ee90 2024-03-29T10:34:54.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.535+0000 7f0d55b13640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d500aac80 con 0x7f0d500af900 2024-03-29T10:34:54.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.535+0000 7f0d55b13640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:54.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.535+0000 7f0d55b13640 10 monclient(hunting): authenticate will time out at 815.798542s 2024-03-29T10:34:54.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.536+0000 7f0d4effd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0d5013ee90 0x7f0d5013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:47727/0 (socket says 172.21.15.117:47727) 2024-03-29T10:34:54.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.536+0000 7f0d4effd640 1 -- 172.21.15.117:0/2817782733 learned_addr learned my addr 172.21.15.117:0/2817782733 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:54.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/2817782733 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453128194 0 0) 0x7f0d500af6f0 con 0x7f0d5013ee90 2024-03-29T10:34:54.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:34:54.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/2817782733 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d38003520 con 0x7f0d5013ee90 2024-03-29T10:34:54.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/2817782733 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2236296573 0 0) 0x7f0d38003520 con 0x7f0d5013ee90 2024-03-29T10:34:54.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/2817782733 >> v1:172.21.15.117:6790/0 conn(0x7f0d500af900 legacy=0x7f0d50149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/2817782733 >> v1:172.21.15.117:6789/0 conn(0x7f0d500a9ac0 legacy=0x7f0d500adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 monclient: found mon.noname-c 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/2817782733 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d500aaf60 con 0x7f0d5013ee90 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:54.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d55b13640 5 monclient: authenticate success, global_id 4255 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d55b13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/2817782733 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0d44001690 con 0x7f0d5013ee90 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: dump: 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:54.528 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:54.529 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:54.529 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:54.529 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:54.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d55b13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:54.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/2817782733 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d440019a0 con 0x7f0d5013ee90 2024-03-29T10:34:54.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:54.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 1 -- 172.21.15.117:0/2817782733 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0d44001d30 con 0x7f0d5013ee90 2024-03-29T10:34:54.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d4e7fc640 10 monclient: dump: 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:54.530 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d55b13640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:54.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.539+0000 7f0d55b13640 4 set_mon_vals no callback set 2024-03-29T10:34:54.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.540+0000 7f0d55b13640 10 monclient: shutdown 2024-03-29T10:34:54.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.540+0000 7f0d55b13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:54.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.540+0000 7f0d55b13640 1 -- 172.21.15.117:0/2817782733 >> v1:172.21.15.136:6789/0 conn(0x7f0d5013ee90 legacy=0x7f0d5013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.541+0000 7f0d55b13640 1 -- 172.21.15.117:0/2817782733 shutdown_connections 2024-03-29T10:34:54.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.541+0000 7f0d55b13640 1 -- 172.21.15.117:0/2817782733 >> 172.21.15.117:0/2817782733 conn(0x7f0d500b35e0 msgr2=0x7f0d500b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:54.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.541+0000 7f0d55b13640 1 -- 172.21.15.117:0/2817782733 shutdown_connections 2024-03-29T10:34:54.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.542+0000 7f0d55b13640 1 -- 172.21.15.117:0/2817782733 wait complete. 2024-03-29T10:34:54.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.543+0000 7f0d55b13640 10 monclient: build_initial_monmap 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.543+0000 7f0d55b13640 10 monclient: monmap: 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.544+0000 7f0d55b13640 1 librados: starting msgr at 2024-03-29T10:34:54.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.544+0000 7f0d55b13640 1 librados: starting objecter 2024-03-29T10:34:54.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.544+0000 7f0d55b13640 1 Processor -- start 2024-03-29T10:34:54.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.544+0000 7f0d55b13640 1 -- start start 2024-03-29T10:34:54.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.544+0000 7f0d55b13640 1 librados: setting wanted keys 2024-03-29T10:34:54.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.545+0000 7f0d55b13640 1 librados: calling monclient init 2024-03-29T10:34:54.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.545+0000 7f0d55b13640 10 monclient: init 2024-03-29T10:34:54.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.547+0000 7f0d55b13640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:54.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.547+0000 7f0d55b13640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:54.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.550+0000 7f0d55b13640 10 monclient(hunting): picked mon.noname-c con 0x7f0d500a9ac0 addr v1:172.21.15.117:6790/0 2024-03-29T10:34:54.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.550+0000 7f0d55b13640 10 monclient(hunting): picked mon.noname-a con 0x7f0d500af900 addr v1:172.21.15.117:6789/0 2024-03-29T10:34:54.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.550+0000 7f0d55b13640 10 monclient(hunting): picked mon.noname-b con 0x7f0d5013ee90 addr v1:172.21.15.136:6789/0 2024-03-29T10:34:54.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.550+0000 7f0d55b13640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d50070150 con 0x7f0d500af900 2024-03-29T10:34:54.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.550+0000 7f0d55b13640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d50070430 con 0x7f0d5013ee90 2024-03-29T10:34:54.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.550+0000 7f0d55b13640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d501395a0 con 0x7f0d500a9ac0 2024-03-29T10:34:54.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.550+0000 7f0d55b13640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:54.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.551+0000 7f0d55b13640 10 monclient(hunting): authenticate will time out at 815.814130s 2024-03-29T10:34:54.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.553+0000 7f0d4f7fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f0d500a9ac0 0x7f0d501d2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:39363/0 (socket says 172.21.15.117:39363) 2024-03-29T10:34:54.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.553+0000 7f0d4f7fe640 1 -- 172.21.15.117:0/4210033601 learned_addr learned my addr 172.21.15.117:0/4210033601 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:34:54.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.553+0000 7f0d37fff640 10 client.?.objecter ms_handle_connect 0x7f0d500a9ac0 2024-03-29T10:34:54.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.553+0000 7f0d37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:54.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.553+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795851562 0 0) 0x7f0d501395a0 con 0x7f0d500a9ac0 2024-03-29T10:34:54.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.553+0000 7f0d37fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.554+0000 7f0d37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.554+0000 7f0d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.554+0000 7f0d37fff640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:34:54.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.554+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d3c004eb0 con 0x7f0d500a9ac0 2024-03-29T10:34:54.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 client.?.objecter ms_handle_connect 0x7f0d5013ee90 2024-03-29T10:34:54.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:54.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1082441424 0 0) 0x7f0d3c004eb0 con 0x7f0d500a9ac0 2024-03-29T10:34:54.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d501395a0 con 0x7f0d500a9ac0 2024-03-29T10:34:54.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640116999 0 0) 0x7f0d50070430 con 0x7f0d5013ee90 2024-03-29T10:34:54.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d3c004eb0 con 0x7f0d5013ee90 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0d40004080 con 0x7f0d500a9ac0 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 monclient(hunting): dump: 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:54.544 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.556+0000 7f0d37fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 10 client.?.objecter ms_handle_connect 0x7f0d500af900 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 990474762 0 0) 0x7f0d3c004eb0 con 0x7f0d5013ee90 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d50070430 con 0x7f0d5013ee90 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0d48001810 con 0x7f0d5013ee90 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 10 monclient(hunting): dump: 2024-03-29T10:34:54.545 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042344348 0 0) 0x7f0d50070150 con 0x7f0d500af900 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:34:54.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.557+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d3c004eb0 con 0x7f0d500af900 2024-03-29T10:34:54.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2578836821 0 0) 0x7f0d3c004eb0 con 0x7f0d500af900 2024-03-29T10:34:54.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d50070150 con 0x7f0d500af900 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0d44001650 con 0x7f0d500af900 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 10 monclient(hunting): dump: 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:54.548 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3336135176 0 0) 0x7f0d50070150 con 0x7f0d500af900 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 >> v1:172.21.15.117:6790/0 conn(0x7f0d500a9ac0 legacy=0x7f0d501d2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 >> v1:172.21.15.136:6789/0 conn(0x7f0d5013ee90 legacy=0x7f0d501de2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 10 monclient: _finish_hunting 0 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 1 monclient: found mon.a 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d50139730 con 0x7f0d500af900 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 10 monclient: _finish_auth 0 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.559+0000 7f0d37fff640 10 monclient: _check_auth_tickets 2024-03-29T10:34:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d55b13640 5 monclient: authenticate success, global_id 4257 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d55b13640 10 monclient: _renew_subs 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d55b13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d501420f0 con 0x7f0d500af900 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d55b13640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d55b13640 10 monclient: _renew_subs 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d55b13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d500722a0 con 0x7f0d500af900 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d55b13640 1 librados: init done 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d44001270 con 0x7f0d500af900 2024-03-29T10:34:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0d44004c30 con 0x7f0d500af900 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d4dffb640 4 set_mon_vals no callback set 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.560+0000 7f0d37fff640 10 monclient: dump: 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:34:54.551 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:34:54.552 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:34:54.552 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:34:54.552 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.564+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f0d4401ac00 con 0x7f0d500af900 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.564+0000 7f0d4f7fe640 10 monclient: get_auth_request con 0x7f0d3c06e0f0 auth_method 0 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.564+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f0d44084ac0 con 0x7f0d500af900 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.564+0000 7f0d37fff640 10 client.4257.objecter ms_dispatch 0x7f0d50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.564+0000 7f0d37fff640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.564+0000 7f0d37fff640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.565+0000 7f0d37fff640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.565+0000 7f0d37fff640 10 client.4257.objecter ms_handle_connect 0x7f0d3c06e0f0 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.566+0000 7f0d55b13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.566+0000 7f0d55b13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.566+0000 7f0d55b13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.566+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d30005080 con 0x7f0d500af900 2024-03-29T10:34:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.577+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d440503a0 con 0x7f0d500af900 2024-03-29T10:34:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.577+0000 7f0d37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.577+0000 7f0d37fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:54.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.895+0000 7f0d55b13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:54.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.896+0000 7f0d55b13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:54.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.896+0000 7f0d55b13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:34:54.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.896+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d30005330 con 0x7f0d500af900 2024-03-29T10:34:54.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.898+0000 7f0d37fff640 1 -- 172.21.15.117:0/4210033601 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f0d440522d0 con 0x7f0d500af900 2024-03-29T10:34:54.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.898+0000 7f0d37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:54.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.898+0000 7f0d37fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:54.885 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:34:54.885 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:34:54.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.902+0000 7f0d55b13640 10 librados: watch_flush enter 2024-03-29T10:34:54.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.902+0000 7f0d55b13640 10 librados: watch_flush exit 2024-03-29T10:34:54.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.902+0000 7f0d55b13640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T10:34:54.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.902+0000 7f0d55b13640 10 client.4257.objecter successfully canceled tick 2024-03-29T10:34:54.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.902+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 >> v1:172.21.15.117:6816/376768171 conn(0x7f0d3c06e0f0 legacy=0x7f0d3c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.903+0000 7f0d55b13640 10 monclient: shutdown 2024-03-29T10:34:54.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.903+0000 7f0d55b13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:54.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.903+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 >> v1:172.21.15.117:6789/0 conn(0x7f0d500af900 legacy=0x7f0d50071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.903+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 shutdown_connections 2024-03-29T10:34:54.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.903+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 >> 172.21.15.117:0/4210033601 conn(0x7f0d500b35e0 msgr2=0x7f0d5013b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:54.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.906+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 shutdown_connections 2024-03-29T10:34:54.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.908+0000 7f0d55b13640 1 -- 172.21.15.117:0/4210033601 wait complete. 2024-03-29T10:34:54.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:34:54.908+0000 7f0d55b13640 1 librados: shutdown 2024-03-29T10:34:54.912 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:01.719 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:01.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.885+0000 7f08da09d640 10 monclient: get_monmap_and_config 2024-03-29T10:35:01.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.885+0000 7f08da09d640 10 monclient: build_initial_monmap 2024-03-29T10:35:01.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.885+0000 7f08da09d640 10 monclient: monmap: 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:35:01.886841+0000 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:35:01.886841+0000 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:35:01.872 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.889+0000 7f08da09d640 1 Processor -- start 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.889+0000 7f08da09d640 1 -- start start 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.889+0000 7f08da09d640 10 monclient: init 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.889+0000 7f08da09d640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.889+0000 7f08da09d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.890+0000 7f08da09d640 10 monclient(hunting): picked mon.noname-b con 0x7f08d40a9930 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.890+0000 7f08da09d640 10 monclient(hunting): picked mon.noname-a con 0x7f08d4149dc0 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.890+0000 7f08da09d640 10 monclient(hunting): picked mon.noname-c con 0x7f08d414ff40 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.890+0000 7f08da09d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d40aae60 con 0x7f08d4149dc0 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.890+0000 7f08da09d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d40b4310 con 0x7f08d414ff40 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.890+0000 7f08da09d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d40afdd0 con 0x7f08d40a9930 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.890+0000 7f08da09d640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:01.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.890+0000 7f08da09d640 10 monclient(hunting): authenticate will time out at 823.152200s 2024-03-29T10:35:01.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.893+0000 7f08d2ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f08d4149dc0 0x7f08d414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:37927/0 (socket says 172.21.15.117:37927) 2024-03-29T10:35:01.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.893+0000 7f08d2ffd640 1 -- 172.21.15.117:0/87272595 learned_addr learned my addr 172.21.15.117:0/87272595 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:01.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.894+0000 7f08d27fc640 1 -- 172.21.15.117:0/87272595 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99289385 0 0) 0x7f08d40aae60 con 0x7f08d4149dc0 2024-03-29T10:35:01.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.894+0000 7f08d27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:01.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.894+0000 7f08d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:01.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.894+0000 7f08d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:01.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.894+0000 7f08d27fc640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:35:01.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.894+0000 7f08d27fc640 1 -- 172.21.15.117:0/87272595 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08bc003520 con 0x7f08d4149dc0 2024-03-29T10:35:01.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.894+0000 7f08d27fc640 1 -- 172.21.15.117:0/87272595 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2237119598 0 0) 0x7f08bc003520 con 0x7f08d4149dc0 2024-03-29T10:35:01.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.894+0000 7f08d27fc640 1 -- 172.21.15.117:0/87272595 >> v1:172.21.15.117:6790/0 conn(0x7f08d40a9930 legacy=0x7f08d4143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:01.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 1 -- 172.21.15.117:0/87272595 >> v1:172.21.15.136:6789/0 conn(0x7f08d414ff40 legacy=0x7f08d4152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:35:01.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:35:01.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 1 monclient: found mon.noname-a 2024-03-29T10:35:01.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:01.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 1 -- 172.21.15.117:0/87272595 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d414a950 con 0x7f08d4149dc0 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: _finish_auth 0 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 1 -- 172.21.15.117:0/87272595 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f08c8001690 con 0x7f08d4149dc0 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: dump: 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:01.886 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08da09d640 5 monclient: authenticate success, global_id 4260 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08da09d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 1 -- 172.21.15.117:0/87272595 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08c8001860 con 0x7f08d4149dc0 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 1 -- 172.21.15.117:0/87272595 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f08c8001cb0 con 0x7f08d4149dc0 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08d27fc640 10 monclient: dump: 2024-03-29T10:35:01.887 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08da09d640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.897+0000 7f08da09d640 4 set_mon_vals no callback set 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.898+0000 7f08da09d640 10 monclient: shutdown 2024-03-29T10:35:01.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.898+0000 7f08da09d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:01.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.898+0000 7f08da09d640 1 -- 172.21.15.117:0/87272595 >> v1:172.21.15.117:6789/0 conn(0x7f08d4149dc0 legacy=0x7f08d414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:01.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.900+0000 7f08da09d640 1 -- 172.21.15.117:0/87272595 shutdown_connections 2024-03-29T10:35:01.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.900+0000 7f08da09d640 1 -- 172.21.15.117:0/87272595 >> 172.21.15.117:0/87272595 conn(0x7f08d40b35e0 msgr2=0x7f08d40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:01.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.903+0000 7f08da09d640 1 -- 172.21.15.117:0/87272595 shutdown_connections 2024-03-29T10:35:01.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.904+0000 7f08da09d640 1 -- 172.21.15.117:0/87272595 wait complete. 2024-03-29T10:35:01.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.905+0000 7f08da09d640 10 monclient: build_initial_monmap 2024-03-29T10:35:01.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.905+0000 7f08da09d640 10 monclient: monmap: 2024-03-29T10:35:01.891 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:01.891 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:01.891 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:35:01.891 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:35:01.891 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:01.891 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:01.891 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:01.892 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:35:01.892 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:35:01.892 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:01.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.906+0000 7f08da09d640 1 librados: starting msgr at 2024-03-29T10:35:01.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.906+0000 7f08da09d640 1 librados: starting objecter 2024-03-29T10:35:01.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.906+0000 7f08da09d640 1 Processor -- start 2024-03-29T10:35:01.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.906+0000 7f08da09d640 1 -- start start 2024-03-29T10:35:01.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.906+0000 7f08da09d640 1 librados: setting wanted keys 2024-03-29T10:35:01.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.906+0000 7f08da09d640 1 librados: calling monclient init 2024-03-29T10:35:01.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.906+0000 7f08da09d640 10 monclient: init 2024-03-29T10:35:01.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.907+0000 7f08da09d640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:01.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.907+0000 7f08da09d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:35:01.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08d37fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f08d40a9930 0x7f08d4071660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:46793/0 (socket says 172.21.15.117:46793) 2024-03-29T10:35:01.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08da09d640 10 monclient(hunting): picked mon.noname-c con 0x7f08d40a9930 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:01.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08da09d640 10 monclient(hunting): picked mon.noname-b con 0x7f08d4149dc0 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:01.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08da09d640 10 monclient(hunting): picked mon.noname-a con 0x7f08d414ff40 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:01.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08da09d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d4070fd0 con 0x7f08d414ff40 2024-03-29T10:35:01.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08da09d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d4072480 con 0x7f08d4149dc0 2024-03-29T10:35:01.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08da09d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d41df5a0 con 0x7f08d40a9930 2024-03-29T10:35:01.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08da09d640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:01.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08da09d640 10 monclient(hunting): authenticate will time out at 823.172890s 2024-03-29T10:35:01.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08d37fe640 1 -- 172.21.15.117:0/3190613176 learned_addr learned my addr 172.21.15.117:0/3190613176 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:01.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08bbfff640 10 client.?.objecter ms_handle_connect 0x7f08d40a9930 2024-03-29T10:35:01.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.910+0000 7f08bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:01.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.914+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447418683 0 0) 0x7f08d41df5a0 con 0x7f08d40a9930 2024-03-29T10:35:01.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.914+0000 7f08bbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:01.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.914+0000 7f08bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:01.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.914+0000 7f08bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:01.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.914+0000 7f08bbfff640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:35:01.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.914+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c0004eb0 con 0x7f08d40a9930 2024-03-29T10:35:01.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.914+0000 7f08bbfff640 10 client.?.objecter ms_handle_connect 0x7f08d4149dc0 2024-03-29T10:35:01.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.914+0000 7f08bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.914+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564925813 0 0) 0x7f08d4072480 con 0x7f08d4149dc0 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d41df5a0 con 0x7f08d4149dc0 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1460159703 0 0) 0x7f08c0004eb0 con 0x7f08d40a9930 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08d4072480 con 0x7f08d40a9930 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f08cc001ce0 con 0x7f08d40a9930 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.915+0000 7f08bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.916+0000 7f08bbfff640 10 monclient(hunting): dump: 2024-03-29T10:35:01.902 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.916+0000 7f08bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.916+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2852175727 0 0) 0x7f08d41df5a0 con 0x7f08d4149dc0 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.916+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08c0004eb0 con 0x7f08d4149dc0 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.916+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2741418071 0 0) 0x7f08d4072480 con 0x7f08d40a9930 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.916+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 >> v1:172.21.15.136:6789/0 conn(0x7f08d4149dc0 legacy=0x7f08d4071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:01.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.916+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 >> v1:172.21.15.117:6789/0 conn(0x7f08d414ff40 legacy=0x7f08d41e6c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:35:01.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08bbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:35:01.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08bbfff640 1 monclient: found mon.c 2024-03-29T10:35:01.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d41df730 con 0x7f08d40a9930 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08bbfff640 10 monclient: _finish_auth 0 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08bbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08da09d640 5 monclient: authenticate success, global_id 4265 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08da09d640 10 monclient: _renew_subs 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08da09d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.917+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d41dc000 con 0x7f08d40a9930 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.918+0000 7f08da09d640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.918+0000 7f08da09d640 10 monclient: _renew_subs 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.918+0000 7f08da09d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.918+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d41deb80 con 0x7f08d40a9930 2024-03-29T10:35:01.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.918+0000 7f08da09d640 1 librados: init done 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08da09d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08da09d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08da09d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d41dbe50 con 0x7f08d40a9930 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08cc003970 con 0x7f08d40a9930 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f08cc003cc0 con 0x7f08d40a9930 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:01.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08bbfff640 10 monclient: dump: 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:01.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.921+0000 7f08d1ffb640 4 set_mon_vals no callback set 2024-03-29T10:35:01.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.926+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f08cc003e20 con 0x7f08d40a9930 2024-03-29T10:35:01.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.927+0000 7f08d2ffd640 10 monclient: get_auth_request con 0x7f08c006dea0 auth_method 0 2024-03-29T10:35:01.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.927+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f08cc084cd0 con 0x7f08d40a9930 2024-03-29T10:35:01.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.927+0000 7f08bbfff640 10 client.4265.objecter ms_dispatch 0x7f08d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:01.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.927+0000 7f08bbfff640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:01.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.927+0000 7f08bbfff640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:01.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.927+0000 7f08bbfff640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T10:35:01.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.927+0000 7f08bbfff640 10 client.4265.objecter ms_handle_connect 0x7f08c006dea0 2024-03-29T10:35:01.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.931+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08cc0505b0 con 0x7f08d40a9930 2024-03-29T10:35:01.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.932+0000 7f08bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:01.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:01.932+0000 7f08bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:02.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.247+0000 7f08da09d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:02.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.247+0000 7f08da09d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:02.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.247+0000 7f08da09d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:02.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.247+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08d41deed0 con 0x7f08d40a9930 2024-03-29T10:35:02.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.250+0000 7f08bbfff640 1 -- 172.21.15.117:0/3190613176 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f08cc0524e0 con 0x7f08d40a9930 2024-03-29T10:35:02.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.250+0000 7f08bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:02.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.250+0000 7f08bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:02.236 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:35:02.236 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:02.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.254+0000 7f08da09d640 10 librados: watch_flush enter 2024-03-29T10:35:02.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.255+0000 7f08da09d640 10 librados: watch_flush exit 2024-03-29T10:35:02.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.255+0000 7f08da09d640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T10:35:02.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.255+0000 7f08da09d640 10 client.4265.objecter successfully canceled tick 2024-03-29T10:35:02.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.256+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 >> v1:172.21.15.117:6816/376768171 conn(0x7f08c006dea0 legacy=0x7f08c0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:02.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.256+0000 7f08da09d640 10 monclient: shutdown 2024-03-29T10:35:02.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.256+0000 7f08da09d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:02.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.256+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 >> v1:172.21.15.117:6790/0 conn(0x7f08d40a9930 legacy=0x7f08d4071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:02.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.256+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 shutdown_connections 2024-03-29T10:35:02.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.257+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 >> 172.21.15.117:0/3190613176 conn(0x7f08d40b35e0 msgr2=0x7f08d40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:02.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.257+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 shutdown_connections 2024-03-29T10:35:02.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.257+0000 7f08da09d640 1 -- 172.21.15.117:0/3190613176 wait complete. 2024-03-29T10:35:02.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:02.257+0000 7f08da09d640 1 librados: shutdown 2024-03-29T10:35:02.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:09.070 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:09.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.236+0000 7fd144bc3640 10 monclient: get_monmap_and_config 2024-03-29T10:35:09.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.236+0000 7fd144bc3640 10 monclient: build_initial_monmap 2024-03-29T10:35:09.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.236+0000 7fd144bc3640 10 monclient: monmap: 2024-03-29T10:35:09.221 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:09.221 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:09.221 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:35:09.237301+0000 2024-03-29T10:35:09.221 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:35:09.237301+0000 2024-03-29T10:35:09.222 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:09.222 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:09.222 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:09.222 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:35:09.222 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:35:09.222 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:09.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.240+0000 7fd144bc3640 1 Processor -- start 2024-03-29T10:35:09.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.240+0000 7fd144bc3640 1 -- start start 2024-03-29T10:35:09.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.240+0000 7fd144bc3640 10 monclient: init 2024-03-29T10:35:09.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:09.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:35:09.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 10 monclient(hunting): picked mon.noname-a con 0x7fd1400b4920 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:09.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 10 monclient(hunting): picked mon.noname-c con 0x7fd1400b4200 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:09.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 10 monclient(hunting): picked mon.noname-b con 0x7fd14013f9a0 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:09.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1400b0930 con 0x7fd1400b4920 2024-03-29T10:35:09.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1400af460 con 0x7fd1400b4200 2024-03-29T10:35:09.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1400aeda0 con 0x7fd14013f9a0 2024-03-29T10:35:09.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.241+0000 7fd144bc3640 10 monclient(hunting): authenticate will time out at 830.503719s 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.245+0000 7fd13ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd14013f9a0 0x7fd14014d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:45397/0 (socket says 172.21.15.117:45397) 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.245+0000 7fd13ffff640 1 -- 172.21.15.117:0/1661115632 learned_addr learned my addr 172.21.15.117:0/1661115632 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.245+0000 7fd13e7fc640 1 -- 172.21.15.117:0/1661115632 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934047314 0 0) 0x7fd1400aeda0 con 0x7fd14013f9a0 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.245+0000 7fd13e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.245+0000 7fd13e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.245+0000 7fd13e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.245+0000 7fd13e7fc640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.245+0000 7fd13e7fc640 1 -- 172.21.15.117:0/1661115632 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd128003520 con 0x7fd14013f9a0 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 1 -- 172.21.15.117:0/1661115632 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215069427 0 0) 0x7fd128003520 con 0x7fd14013f9a0 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 1 -- 172.21.15.117:0/1661115632 >> v1:172.21.15.136:6789/0 conn(0x7fd1400b4200 legacy=0x7fd1400b0fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:09.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 1 -- 172.21.15.117:0/1661115632 >> v1:172.21.15.117:6789/0 conn(0x7fd1400b4920 legacy=0x7fd14013f290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:09.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:35:09.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:35:09.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:09.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:35:09.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 1 -- 172.21.15.117:0/1661115632 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1400af980 con 0x7fd14013f9a0 2024-03-29T10:35:09.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd144bc3640 5 monclient: authenticate success, global_id 4268 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd144bc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 1 -- 172.21.15.117:0/1661115632 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd138001a10 con 0x7fd14013f9a0 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 10 monclient: dump: 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:09.235 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd144bc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 1 -- 172.21.15.117:0/1661115632 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd138001ca0 con 0x7fd14013f9a0 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd13e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd144bc3640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.246+0000 7fd144bc3640 4 set_mon_vals no callback set 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.247+0000 7fd144bc3640 10 monclient: shutdown 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.247+0000 7fd144bc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.247+0000 7fd144bc3640 1 -- 172.21.15.117:0/1661115632 >> v1:172.21.15.117:6790/0 conn(0x7fd14013f9a0 legacy=0x7fd14014d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.247+0000 7fd144bc3640 1 -- 172.21.15.117:0/1661115632 shutdown_connections 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.247+0000 7fd144bc3640 1 -- 172.21.15.117:0/1661115632 >> 172.21.15.117:0/1661115632 conn(0x7fd14006d230 msgr2=0x7fd140066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:09.236 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.249+0000 7fd144bc3640 1 -- 172.21.15.117:0/1661115632 shutdown_connections 2024-03-29T10:35:09.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.249+0000 7fd144bc3640 1 -- 172.21.15.117:0/1661115632 wait complete. 2024-03-29T10:35:09.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.251+0000 7fd144bc3640 10 monclient: build_initial_monmap 2024-03-29T10:35:09.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.251+0000 7fd144bc3640 10 monclient: monmap: 2024-03-29T10:35:09.238 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:09.238 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:09.238 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:35:09.238 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:35:09.238 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:09.238 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.251+0000 7fd144bc3640 1 librados: starting msgr at 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.251+0000 7fd144bc3640 1 librados: starting objecter 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.251+0000 7fd144bc3640 1 Processor -- start 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.251+0000 7fd144bc3640 1 -- start start 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.251+0000 7fd144bc3640 1 librados: setting wanted keys 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.251+0000 7fd144bc3640 1 librados: calling monclient init 2024-03-29T10:35:09.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.251+0000 7fd144bc3640 10 monclient: init 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 10 monclient(hunting): picked mon.noname-b con 0x7fd1400b4200 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 10 monclient(hunting): picked mon.noname-c con 0x7fd1400b4920 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 10 monclient(hunting): picked mon.noname-a con 0x7fd14013f9a0 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1400ade70 con 0x7fd14013f9a0 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1401db0b0 con 0x7fd1400b4200 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1401db240 con 0x7fd1400b4920 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:09.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.252+0000 7fd144bc3640 10 monclient(hunting): authenticate will time out at 830.514931s 2024-03-29T10:35:09.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd13ffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd14013f9a0 0x7fd1401e1990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:47965/0 (socket says 172.21.15.117:47965) 2024-03-29T10:35:09.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd13ffff640 1 -- 172.21.15.117:0/3617452977 learned_addr learned my addr 172.21.15.117:0/3617452977 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:09.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd127fff640 10 client.?.objecter ms_handle_connect 0x7fd14013f9a0 2024-03-29T10:35:09.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd127fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:09.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506721480 0 0) 0x7fd1400ade70 con 0x7fd14013f9a0 2024-03-29T10:35:09.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd127fff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:09.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd127fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:09.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:09.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd127fff640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:35:09.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.256+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd12c004eb0 con 0x7fd14013f9a0 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.257+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 139481410 0 0) 0x7fd12c004eb0 con 0x7fd14013f9a0 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.257+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd12c005300 con 0x7fd14013f9a0 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 client.?.objecter ms_handle_connect 0x7fd1400b4920 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd138004d00 con 0x7fd14013f9a0 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 monclient(hunting): dump: 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:09.245 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3366941014 0 0) 0x7fd12c005300 con 0x7fd14013f9a0 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 >> v1:172.21.15.117:6790/0 conn(0x7fd1400b4920 legacy=0x7fd1400ad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 >> v1:172.21.15.136:6789/0 conn(0x7fd1400b4200 legacy=0x7fd1401da630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 monclient: _finish_hunting 0 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 1 monclient: found mon.a 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd140145ac0 con 0x7fd14013f9a0 2024-03-29T10:35:09.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 monclient: _finish_auth 0 2024-03-29T10:35:09.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 10 monclient: _check_auth_tickets 2024-03-29T10:35:09.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:09.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd144bc3640 5 monclient: authenticate success, global_id 4263 2024-03-29T10:35:09.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd144bc3640 10 monclient: _renew_subs 2024-03-29T10:35:09.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd144bc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.258+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd140145d90 con 0x7fd14013f9a0 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.259+0000 7fd144bc3640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.259+0000 7fd144bc3640 10 monclient: _renew_subs 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.259+0000 7fd144bc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.259+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd140140040 con 0x7fd14013f9a0 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.259+0000 7fd144bc3640 1 librados: init done 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.260+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd138001e90 con 0x7fd14013f9a0 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.260+0000 7fd127fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.260+0000 7fd13d7fa640 4 set_mon_vals no callback set 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.260+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd138004d00 con 0x7fd14013f9a0 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.260+0000 7fd127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.260+0000 7fd127fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.260+0000 7fd127fff640 10 monclient: dump: 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:09.250 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.262+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fd138005190 con 0x7fd14013f9a0 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.263+0000 7fd13f7fe640 10 monclient: get_auth_request con 0x7fd12c06de70 auth_method 0 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.263+0000 7fd127fff640 10 client.4263.objecter ms_handle_connect 0x7fd12c06de70 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.263+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fd138054b00 con 0x7fd14013f9a0 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.263+0000 7fd127fff640 10 client.4263.objecter ms_dispatch 0x7fd140000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.263+0000 7fd127fff640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:09.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.263+0000 7fd127fff640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:09.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.263+0000 7fd127fff640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T10:35:09.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.265+0000 7fd144bc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:09.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.265+0000 7fd144bc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:09.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.265+0000 7fd144bc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:09.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.265+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd120005080 con 0x7fd14013f9a0 2024-03-29T10:35:09.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.276+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd138050da0 con 0x7fd14013f9a0 2024-03-29T10:35:09.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.276+0000 7fd127fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:09.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.276+0000 7fd127fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:09.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.600+0000 7fd144bc3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:09.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.600+0000 7fd144bc3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:09.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.600+0000 7fd144bc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:09.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.600+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd120005330 con 0x7fd14013f9a0 2024-03-29T10:35:09.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.601+0000 7fd127fff640 1 -- 172.21.15.117:0/3617452977 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fd138050da0 con 0x7fd14013f9a0 2024-03-29T10:35:09.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.601+0000 7fd127fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:09.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.601+0000 7fd127fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:09.590 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:35:09.590 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.605+0000 7fd144bc3640 10 librados: watch_flush enter 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.605+0000 7fd144bc3640 10 librados: watch_flush exit 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.605+0000 7fd144bc3640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.605+0000 7fd144bc3640 10 client.4263.objecter successfully canceled tick 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.606+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 >> v1:172.21.15.117:6816/376768171 conn(0x7fd12c06de70 legacy=0x7fd12c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.606+0000 7fd144bc3640 10 monclient: shutdown 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.606+0000 7fd144bc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.606+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 >> v1:172.21.15.117:6789/0 conn(0x7fd14013f9a0 legacy=0x7fd1401e1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.606+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 shutdown_connections 2024-03-29T10:35:09.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.606+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 >> 172.21.15.117:0/3617452977 conn(0x7fd14006d230 msgr2=0x7fd1400b6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:09.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.608+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 shutdown_connections 2024-03-29T10:35:09.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.608+0000 7fd144bc3640 1 -- 172.21.15.117:0/3617452977 wait complete. 2024-03-29T10:35:09.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:09.608+0000 7fd144bc3640 1 librados: shutdown 2024-03-29T10:35:09.612 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:16.420 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:16.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.594+0000 7fb9b27d0640 10 monclient: get_monmap_and_config 2024-03-29T10:35:16.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.594+0000 7fb9b27d0640 10 monclient: build_initial_monmap 2024-03-29T10:35:16.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.594+0000 7fb9b27d0640 10 monclient: monmap: 2024-03-29T10:35:16.582 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:16.582 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:16.582 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:35:16.595285+0000 2024-03-29T10:35:16.582 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:35:16.595285+0000 2024-03-29T10:35:16.582 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:16.582 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:16.582 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:16.583 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:35:16.583 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:35:16.583 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:16.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.596+0000 7fb9b27d0640 1 Processor -- start 2024-03-29T10:35:16.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.597+0000 7fb9b27d0640 1 -- start start 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.597+0000 7fb9b27d0640 10 monclient: init 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ac1422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ac142690 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ac0b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac142c80 con 0x7fb9ac142690 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac0aae40 con 0x7fb9ac1422b0 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac13fbd0 con 0x7fb9ac0b17d0 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.598+0000 7fb9b27d0640 10 monclient(hunting): authenticate will time out at 837.860499s 2024-03-29T10:35:16.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.601+0000 7fb9ab7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fb9ac142690 0x7fb9ac0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:54119/0 (socket says 172.21.15.117:54119) 2024-03-29T10:35:16.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.601+0000 7fb9ab7fe640 1 -- 172.21.15.117:0/1449769023 learned_addr learned my addr 172.21.15.117:0/1449769023 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.602+0000 7fb9aaffd640 1 -- 172.21.15.117:0/1449769023 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467367395 0 0) 0x7fb9ac142c80 con 0x7fb9ac142690 2024-03-29T10:35:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.602+0000 7fb9aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:35:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.602+0000 7fb9aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.602+0000 7fb9aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.602+0000 7fb9aaffd640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:35:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.602+0000 7fb9aaffd640 1 -- 172.21.15.117:0/1449769023 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb994003520 con 0x7fb9ac142690 2024-03-29T10:35:16.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 1 -- 172.21.15.117:0/1449769023 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1140461401 0 0) 0x7fb994003520 con 0x7fb9ac142690 2024-03-29T10:35:16.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 1 -- 172.21.15.117:0/1449769023 >> v1:172.21.15.117:6790/0 conn(0x7fb9ac0b17d0 legacy=0x7fb9ac0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 1 -- 172.21.15.117:0/1449769023 >> v1:172.21.15.136:6789/0 conn(0x7fb9ac1422b0 legacy=0x7fb9ac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 1 monclient: found mon.noname-a 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 1 -- 172.21.15.117:0/1449769023 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac151940 con 0x7fb9ac142690 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 10 monclient: _finish_auth 0 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 1 -- 172.21.15.117:0/1449769023 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fb99c001670 con 0x7fb9ac142690 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9aaffd640 10 monclient: dump: 2024-03-29T10:35:16.591 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9b27d0640 5 monclient: authenticate success, global_id 4266 2024-03-29T10:35:16.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.605+0000 7fb9b27d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:16.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.607+0000 7fb9aaffd640 1 -- 172.21.15.117:0/1449769023 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb99c0018d0 con 0x7fb9ac142690 2024-03-29T10:35:16.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.607+0000 7fb9aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:16.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.607+0000 7fb9aaffd640 1 -- 172.21.15.117:0/1449769023 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fb99c001ca0 con 0x7fb9ac142690 2024-03-29T10:35:16.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.607+0000 7fb9aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:16.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.607+0000 7fb9aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:35:16.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.607+0000 7fb9aaffd640 10 monclient: dump: 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:16.595 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.607+0000 7fb9b27d0640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.607+0000 7fb9b27d0640 4 set_mon_vals no callback set 2024-03-29T10:35:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.608+0000 7fb9b27d0640 10 monclient: shutdown 2024-03-29T10:35:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.608+0000 7fb9b27d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.608+0000 7fb9b27d0640 1 -- 172.21.15.117:0/1449769023 >> v1:172.21.15.117:6789/0 conn(0x7fb9ac142690 legacy=0x7fb9ac0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.608+0000 7fb9b27d0640 1 -- 172.21.15.117:0/1449769023 shutdown_connections 2024-03-29T10:35:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.608+0000 7fb9b27d0640 1 -- 172.21.15.117:0/1449769023 >> 172.21.15.117:0/1449769023 conn(0x7fb9ac066a00 msgr2=0x7fb9ac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.609+0000 7fb9b27d0640 1 -- 172.21.15.117:0/1449769023 shutdown_connections 2024-03-29T10:35:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.609+0000 7fb9b27d0640 1 -- 172.21.15.117:0/1449769023 wait complete. 2024-03-29T10:35:16.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.610+0000 7fb9b27d0640 10 monclient: build_initial_monmap 2024-03-29T10:35:16.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.610+0000 7fb9b27d0640 10 monclient: monmap: 2024-03-29T10:35:16.598 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:16.598 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:16.598 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:35:16.598 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:35:16.598 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:16.598 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.611+0000 7fb9b27d0640 1 librados: starting msgr at 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.611+0000 7fb9b27d0640 1 librados: starting objecter 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.611+0000 7fb9b27d0640 1 Processor -- start 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.611+0000 7fb9b27d0640 1 -- start start 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.611+0000 7fb9b27d0640 1 librados: setting wanted keys 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.611+0000 7fb9b27d0640 1 librados: calling monclient init 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.611+0000 7fb9b27d0640 10 monclient: init 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.612+0000 7fb9b27d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.612+0000 7fb9b27d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.612+0000 7fb9b27d0640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ac0b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.613+0000 7fb9b27d0640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ac1422b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:16.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.613+0000 7fb9b27d0640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ac142690 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:16.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.616+0000 7fb9b27d0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac12dba0 con 0x7fb9ac0b17d0 2024-03-29T10:35:16.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.616+0000 7fb9b27d0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac12de80 con 0x7fb9ac142690 2024-03-29T10:35:16.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.616+0000 7fb9b27d0640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac06e010 con 0x7fb9ac1422b0 2024-03-29T10:35:16.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.616+0000 7fb9b27d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:16.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.616+0000 7fb9b27d0640 10 monclient(hunting): authenticate will time out at 837.878340s 2024-03-29T10:35:16.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.619+0000 7fb9b0d6d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb9ac142690 0x7fb9ac1e6740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:57113/0 (socket says 172.21.15.117:57113) 2024-03-29T10:35:16.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.620+0000 7fb9b0d6d640 1 -- 172.21.15.117:0/330323850 learned_addr learned my addr 172.21.15.117:0/330323850 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:16.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.620+0000 7fb9a8ff9640 10 client.?.objecter ms_handle_connect 0x7fb9ac1422b0 2024-03-29T10:35:16.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.620+0000 7fb9a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:16.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.620+0000 7fb9a8ff9640 10 client.?.objecter ms_handle_connect 0x7fb9ac142690 2024-03-29T10:35:16.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.620+0000 7fb9a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:16.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.620+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582580837 0 0) 0x7fb9ac06e010 con 0x7fb9ac1422b0 2024-03-29T10:35:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:35:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:35:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb998004eb0 con 0x7fb9ac1422b0 2024-03-29T10:35:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482488170 0 0) 0x7fb9ac12de80 con 0x7fb9ac142690 2024-03-29T10:35:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:35:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:35:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac06e010 con 0x7fb9ac142690 2024-03-29T10:35:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1962573868 0 0) 0x7fb998004eb0 con 0x7fb9ac1422b0 2024-03-29T10:35:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9ac12de80 con 0x7fb9ac1422b0 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fb99c001be0 con 0x7fb9ac1422b0 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): dump: 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:16.611 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.623+0000 7fb9a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.624+0000 7fb9a8ff9640 10 client.?.objecter ms_handle_connect 0x7fb9ac0b17d0 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.624+0000 7fb9a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.624+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987622097 0 0) 0x7fb9ac12dba0 con 0x7fb9ac0b17d0 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.624+0000 7fb9a8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.624+0000 7fb9a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.624+0000 7fb9a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.624+0000 7fb9a8ff9640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:35:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.624+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb998004eb0 con 0x7fb9ac0b17d0 2024-03-29T10:35:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 69190878 0 0) 0x7fb9ac06e010 con 0x7fb9ac142690 2024-03-29T10:35:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9ac12dba0 con 0x7fb9ac142690 2024-03-29T10:35:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fb9a0004000 con 0x7fb9ac142690 2024-03-29T10:35:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:35:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 10 monclient(hunting): dump: 2024-03-29T10:35:16.613 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:16.613 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 349604779 0 0) 0x7fb9ac12de80 con 0x7fb9ac1422b0 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 >> v1:172.21.15.136:6789/0 conn(0x7fb9ac142690 legacy=0x7fb9ac1e6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 >> v1:172.21.15.117:6789/0 conn(0x7fb9ac0b17d0 legacy=0x7fb9ac06e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:35:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 1 monclient: found mon.c 2024-03-29T10:35:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac12cc70 con 0x7fb9ac1422b0 2024-03-29T10:35:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:35:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:35:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.625+0000 7fb9a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:16.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.627+0000 7fb9b27d0640 5 monclient: authenticate success, global_id 4274 2024-03-29T10:35:16.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.627+0000 7fb9b27d0640 10 monclient: _renew_subs 2024-03-29T10:35:16.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.627+0000 7fb9b27d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:16.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.627+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ac12c890 con 0x7fb9ac1422b0 2024-03-29T10:35:16.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.627+0000 7fb9b27d0640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:16.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.627+0000 7fb9b27d0640 10 monclient: _renew_subs 2024-03-29T10:35:16.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.627+0000 7fb9b27d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:16.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.627+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ac12d370 con 0x7fb9ac1422b0 2024-03-29T10:35:16.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.627+0000 7fb9b27d0640 1 librados: init done 2024-03-29T10:35:16.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.628+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb99c004a90 con 0x7fb9ac1422b0 2024-03-29T10:35:16.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.628+0000 7fb9a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:16.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.628+0000 7fb9a9ffb640 4 set_mon_vals no callback set 2024-03-29T10:35:16.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.628+0000 7fb9b27d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:16.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.629+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fb99c004da0 con 0x7fb9ac1422b0 2024-03-29T10:35:16.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.629+0000 7fb9a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:16.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.629+0000 7fb9a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:16.619 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.629+0000 7fb9a8ff9640 10 monclient: dump: 2024-03-29T10:35:16.619 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:16.619 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:16.619 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.619 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:16.619 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:16.619 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:16.620 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:16.620 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:16.620 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:16.620 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:16.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.629+0000 7fb9b27d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:16.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.629+0000 7fb9b27d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:16.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.629+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ac075f10 con 0x7fb9ac1422b0 2024-03-29T10:35:16.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.630+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fb99c01ad70 con 0x7fb9ac1422b0 2024-03-29T10:35:16.621 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.634+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fb99c084a90 con 0x7fb9ac1422b0 2024-03-29T10:35:16.621 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.635+0000 7fb9a8ff9640 10 client.4274.objecter ms_dispatch 0x7fb9ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:16.621 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.635+0000 7fb9a8ff9640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:16.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.635+0000 7fb9a8ff9640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:16.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.635+0000 7fb9a8ff9640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T10:35:16.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.635+0000 7fb9abfff640 10 monclient: get_auth_request con 0x7fb99806dde0 auth_method 0 2024-03-29T10:35:16.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.636+0000 7fb9a8ff9640 10 client.4274.objecter ms_handle_connect 0x7fb99806dde0 2024-03-29T10:35:16.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.640+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb99c050370 con 0x7fb9ac1422b0 2024-03-29T10:35:16.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.640+0000 7fb9a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:16.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.640+0000 7fb9a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:16.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.965+0000 7fb9b27d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:16.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.965+0000 7fb9b27d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:16.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.965+0000 7fb9b27d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.965+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9ac12ce00 con 0x7fb9ac1422b0 2024-03-29T10:35:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.966+0000 7fb9a8ff9640 1 -- 172.21.15.117:0/330323850 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fb99c001d40 con 0x7fb9ac1422b0 2024-03-29T10:35:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.966+0000 7fb9a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.966+0000 7fb9a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:16.953 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:35:16.953 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:16.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.971+0000 7fb9b27d0640 10 librados: watch_flush enter 2024-03-29T10:35:16.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.971+0000 7fb9b27d0640 10 librados: watch_flush exit 2024-03-29T10:35:16.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.971+0000 7fb9b27d0640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T10:35:16.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.971+0000 7fb9b27d0640 10 client.4274.objecter successfully canceled tick 2024-03-29T10:35:16.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.972+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 >> v1:172.21.15.117:6816/376768171 conn(0x7fb99806dde0 legacy=0x7fb9980702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.972+0000 7fb9b27d0640 10 monclient: shutdown 2024-03-29T10:35:16.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.972+0000 7fb9b27d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:16.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.972+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 >> v1:172.21.15.117:6790/0 conn(0x7fb9ac1422b0 legacy=0x7fb9ac1e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.972+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 shutdown_connections 2024-03-29T10:35:16.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.972+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 >> 172.21.15.117:0/330323850 conn(0x7fb9ac066a00 msgr2=0x7fb9ac13db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:16.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.972+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 shutdown_connections 2024-03-29T10:35:16.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.972+0000 7fb9b27d0640 1 -- 172.21.15.117:0/330323850 wait complete. 2024-03-29T10:35:16.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:16.972+0000 7fb9b27d0640 1 librados: shutdown 2024-03-29T10:35:16.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:23.785 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:23.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.952+0000 7f4e39f2d640 10 monclient: get_monmap_and_config 2024-03-29T10:35:23.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.952+0000 7f4e39f2d640 10 monclient: build_initial_monmap 2024-03-29T10:35:23.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.952+0000 7f4e39f2d640 10 monclient: monmap: 2024-03-29T10:35:23.937 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:23.937 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:23.937 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:35:23.953034+0000 2024-03-29T10:35:23.937 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:35:23.953034+0000 2024-03-29T10:35:23.937 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:23.938 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:23.938 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:23.938 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:35:23.938 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:35:23.938 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.955+0000 7f4e39f2d640 1 Processor -- start 2024-03-29T10:35:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.956+0000 7f4e39f2d640 1 -- start start 2024-03-29T10:35:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.956+0000 7f4e39f2d640 10 monclient: init 2024-03-29T10:35:23.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.957+0000 7f4e39f2d640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:23.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.957+0000 7f4e39f2d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:35:23.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.957+0000 7f4e39f2d640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2c006a70 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.962+0000 7f4e39f2d640 10 monclient(hunting): picked mon.noname-c con 0x7f4e2c02b100 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:23.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.963+0000 7f4e38f2b640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f4e2c006a70 0x7f4e2c0c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:36285/0 (socket says 172.21.15.117:36285) 2024-03-29T10:35:23.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.964+0000 7f4e39f2d640 10 monclient(hunting): picked mon.noname-b con 0x7f4e2c029c30 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:23.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.964+0000 7f4e39f2d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c02a270 con 0x7f4e2c006a70 2024-03-29T10:35:23.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.964+0000 7f4e39f2d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c02c440 con 0x7f4e2c02b100 2024-03-29T10:35:23.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.964+0000 7f4e39f2d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0c9440 con 0x7f4e2c029c30 2024-03-29T10:35:23.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.964+0000 7f4e39f2d640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:23.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.964+0000 7f4e39f2d640 10 monclient(hunting): authenticate will time out at 845.226323s 2024-03-29T10:35:23.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.964+0000 7f4e38f2b640 1 -- 172.21.15.117:0/3763593535 learned_addr learned my addr 172.21.15.117:0/3763593535 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:23.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101141684 0 0) 0x7f4e2c02a270 con 0x7f4e2c006a70 2024-03-29T10:35:23.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:35:23.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:23.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:23.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:35:23.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e24003520 con 0x7f4e2c006a70 2024-03-29T10:35:23.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551156037 0 0) 0x7f4e2c0c9440 con 0x7f4e2c029c30 2024-03-29T10:35:23.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:35:23.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:23.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:23.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:35:23.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.967+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e2c02a270 con 0x7f4e2c029c30 2024-03-29T10:35:23.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3051364557 0 0) 0x7f4e2c02a270 con 0x7f4e2c029c30 2024-03-29T10:35:23.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 >> v1:172.21.15.136:6789/0 conn(0x7f4e2c02b100 legacy=0x7f4e2c0cf100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 >> v1:172.21.15.117:6789/0 conn(0x7f4e2c006a70 legacy=0x7f4e2c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 10 monclient: _finish_hunting 0 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 1 monclient: found mon.noname-b 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c0c95d0 con 0x7f4e2c029c30 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 10 monclient: _finish_auth 0 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 10 monclient: _check_auth_tickets 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4e3404f730 con 0x7f4e2c029c30 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:23.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 10 monclient: dump: 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e39f2d640 5 monclient: authenticate success, global_id 4277 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e39f2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e340008d0 con 0x7f4e2c029c30 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:23.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e39f2d640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:23.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.968+0000 7f4e39f2d640 4 set_mon_vals no callback set 2024-03-29T10:35:23.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.969+0000 7f4e39f2d640 10 monclient: shutdown 2024-03-29T10:35:23.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.969+0000 7f4e39f2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:23.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.969+0000 7f4e39f2d640 1 -- 172.21.15.117:0/3763593535 >> v1:172.21.15.117:6790/0 conn(0x7f4e2c029c30 legacy=0x7f4e2c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:23.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.969+0000 7f4e337fe640 1 -- 172.21.15.117:0/3763593535 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4e340758b0 con 0x7f4e2c029c30 2024-03-29T10:35:23.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.969+0000 7f4e337fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:35:23.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.969+0000 7f4e39f2d640 1 -- 172.21.15.117:0/3763593535 shutdown_connections 2024-03-29T10:35:23.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.969+0000 7f4e39f2d640 1 -- 172.21.15.117:0/3763593535 >> 172.21.15.117:0/3763593535 conn(0x7f4e2c0058f0 msgr2=0x7f4e2c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:23.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.970+0000 7f4e39f2d640 1 -- 172.21.15.117:0/3763593535 shutdown_connections 2024-03-29T10:35:23.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.972+0000 7f4e39f2d640 1 -- 172.21.15.117:0/3763593535 wait complete. 2024-03-29T10:35:23.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.973+0000 7f4e39f2d640 10 monclient: build_initial_monmap 2024-03-29T10:35:23.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.973+0000 7f4e39f2d640 10 monclient: monmap: 2024-03-29T10:35:23.958 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:23.958 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:23.958 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:35:23.958 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:35:23.958 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:23.959 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:23.959 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:23.959 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:35:23.959 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:35:23.959 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:23.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.973+0000 7f4e39f2d640 1 librados: starting msgr at 2024-03-29T10:35:23.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.973+0000 7f4e39f2d640 1 librados: starting objecter 2024-03-29T10:35:23.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.973+0000 7f4e39f2d640 1 Processor -- start 2024-03-29T10:35:23.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.975+0000 7f4e39f2d640 1 -- start start 2024-03-29T10:35:23.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.975+0000 7f4e39f2d640 1 librados: setting wanted keys 2024-03-29T10:35:23.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.975+0000 7f4e39f2d640 1 librados: calling monclient init 2024-03-29T10:35:23.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.975+0000 7f4e39f2d640 10 monclient: init 2024-03-29T10:35:23.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.976+0000 7f4e39f2d640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:23.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.976+0000 7f4e39f2d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:35:23.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.976+0000 7f4e39f2d640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2c006a70 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:23.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.976+0000 7f4e39f2d640 10 monclient(hunting): picked mon.noname-c con 0x7f4e2c029c30 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:23.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.978+0000 7f4e39f2d640 10 monclient(hunting): picked mon.noname-b con 0x7f4e2c0cca80 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:23.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.978+0000 7f4e39f2d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c02f190 con 0x7f4e2c006a70 2024-03-29T10:35:23.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.978+0000 7f4e39f2d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c161670 con 0x7f4e2c0cca80 2024-03-29T10:35:23.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.978+0000 7f4e39f2d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c161800 con 0x7f4e2c029c30 2024-03-29T10:35:23.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.978+0000 7f4e39f2d640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:23.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.978+0000 7f4e39f2d640 10 monclient(hunting): authenticate will time out at 845.240940s 2024-03-29T10:35:23.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.980+0000 7f4e38f2b640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f4e2c006a70 0x7f4e2c02d990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:36965/0 (socket says 172.21.15.117:36965) 2024-03-29T10:35:23.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.981+0000 7f4e38f2b640 1 -- 172.21.15.117:0/720656450 learned_addr learned my addr 172.21.15.117:0/720656450 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.981+0000 7f4e317fa640 10 client.?.objecter ms_handle_connect 0x7f4e2c006a70 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.981+0000 7f4e317fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.984+0000 7f4e317fa640 10 client.?.objecter ms_handle_connect 0x7f4e2c029c30 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.984+0000 7f4e317fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.984+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766799693 0 0) 0x7f4e2c02f190 con 0x7f4e2c006a70 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.984+0000 7f4e317fa640 10 monclient(hunting): _negotiate 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.984+0000 7f4e317fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.984+0000 7f4e317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.984+0000 7f4e317fa640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.984+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e18004eb0 con 0x7f4e2c006a70 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.985+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642442336 0 0) 0x7f4e2c161800 con 0x7f4e2c029c30 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient(hunting): _negotiate 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:23.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e2c02f190 con 0x7f4e2c029c30 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 249306964 0 0) 0x7f4e2c02f190 con 0x7f4e2c029c30 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e2c161800 con 0x7f4e2c029c30 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4e28001790 con 0x7f4e2c029c30 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient(hunting): dump: 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:23.976 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 961129254 0 0) 0x7f4e2c161800 con 0x7f4e2c029c30 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 >> v1:172.21.15.136:6789/0 conn(0x7f4e2c0cca80 legacy=0x7f4e2c0c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 >> v1:172.21.15.117:6789/0 conn(0x7f4e2c006a70 legacy=0x7f4e2c02d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient: _finish_hunting 0 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 1 monclient: found mon.c 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c0c2e20 con 0x7f4e2c029c30 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient: _finish_auth 0 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 10 monclient: _check_auth_tickets 2024-03-29T10:35:23.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e39f2d640 5 monclient: authenticate success, global_id 4280 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e39f2d640 10 monclient: _renew_subs 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.986+0000 7f4e39f2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c0c3010 con 0x7f4e2c029c30 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e39f2d640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e39f2d640 10 monclient: _renew_subs 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e39f2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c02cf80 con 0x7f4e2c029c30 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e39f2d640 1 librados: init done 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e28001d20 con 0x7f4e2c029c30 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e327fc640 4 set_mon_vals no callback set 2024-03-29T10:35:23.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4e28003790 con 0x7f4e2c029c30 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.987+0000 7f4e317fa640 10 monclient: dump: 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:23.979 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.990+0000 7f4e39f2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.990+0000 7f4e39f2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.990+0000 7f4e39f2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.990+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e14005080 con 0x7f4e2c029c30 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.992+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f4e280038f0 con 0x7f4e2c029c30 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.993+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f4e28083720 con 0x7f4e2c029c30 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.993+0000 7f4e317fa640 10 client.4280.objecter ms_dispatch 0x7f4e34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.993+0000 7f4e317fa640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.993+0000 7f4e317fa640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.993+0000 7f4e317fa640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.993+0000 7f4e38f2b640 10 monclient: get_auth_request con 0x7f4e1806da50 auth_method 0 2024-03-29T10:35:23.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:23.994+0000 7f4e317fa640 10 client.4280.objecter ms_handle_connect 0x7f4e1806da50 2024-03-29T10:35:23.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.001+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e2804f000 con 0x7f4e2c029c30 2024-03-29T10:35:23.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.001+0000 7f4e317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:23.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.001+0000 7f4e317fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:24.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.319+0000 7f4e39f2d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:24.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.319+0000 7f4e39f2d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:24.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.319+0000 7f4e39f2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:24.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.319+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e14005330 con 0x7f4e2c029c30 2024-03-29T10:35:24.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.320+0000 7f4e317fa640 1 -- 172.21.15.117:0/720656450 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f4e28050f30 con 0x7f4e2c029c30 2024-03-29T10:35:24.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.320+0000 7f4e317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:24.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.320+0000 7f4e317fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:24.305 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:35:24.306 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:24.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.323+0000 7f4e39f2d640 10 librados: watch_flush enter 2024-03-29T10:35:24.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.323+0000 7f4e39f2d640 10 librados: watch_flush exit 2024-03-29T10:35:24.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.323+0000 7f4e39f2d640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T10:35:24.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.323+0000 7f4e39f2d640 10 client.4280.objecter successfully canceled tick 2024-03-29T10:35:24.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.326+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 >> v1:172.21.15.117:6816/376768171 conn(0x7f4e1806da50 legacy=0x7f4e1806ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:24.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.326+0000 7f4e39f2d640 10 monclient: shutdown 2024-03-29T10:35:24.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.326+0000 7f4e39f2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:24.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.326+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 >> v1:172.21.15.117:6790/0 conn(0x7f4e2c029c30 legacy=0x7f4e2c160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:24.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.326+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 shutdown_connections 2024-03-29T10:35:24.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.326+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 >> 172.21.15.117:0/720656450 conn(0x7f4e2c0058f0 msgr2=0x7f4e2c032c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:24.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.327+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 shutdown_connections 2024-03-29T10:35:24.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.327+0000 7f4e39f2d640 1 -- 172.21.15.117:0/720656450 wait complete. 2024-03-29T10:35:24.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:24.327+0000 7f4e39f2d640 1 librados: shutdown 2024-03-29T10:35:24.330 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:31.137 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.297+0000 7f77bcf42640 10 monclient: get_monmap_and_config 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.297+0000 7f77bcf42640 10 monclient: build_initial_monmap 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.297+0000 7f77bcf42640 10 monclient: monmap: 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:35:31.298824+0000 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:35:31.298824+0000 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:31.283 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:35:31.284 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:35:31.284 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:31.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.298+0000 7f77bcf42640 1 Processor -- start 2024-03-29T10:35:31.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.299+0000 7f77bcf42640 1 -- start start 2024-03-29T10:35:31.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.299+0000 7f77bcf42640 10 monclient: init 2024-03-29T10:35:31.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.299+0000 7f77bcf42640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:31.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.299+0000 7f77bcf42640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:35:31.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.299+0000 7f77bcf42640 10 monclient(hunting): picked mon.noname-c con 0x7f77b80b2800 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:31.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77bcf42640 10 monclient(hunting): picked mon.noname-a con 0x7f77b8142b20 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:31.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77bcf42640 10 monclient(hunting): picked mon.noname-b con 0x7f77b813c040 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:31.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77bcf42640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b80aad10 con 0x7f77b8142b20 2024-03-29T10:35:31.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77bcf42640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b8141f30 con 0x7f77b80b2800 2024-03-29T10:35:31.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77bcf42640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b80aaf60 con 0x7f77b813c040 2024-03-29T10:35:31.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77bcf42640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:31.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77bcf42640 10 monclient(hunting): authenticate will time out at 852.562184s 2024-03-29T10:35:31.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77b6ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f77b8142b20 0x7f77b813b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:53012/0 (socket says 172.21.15.117:53012) 2024-03-29T10:35:31.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77b6ffd640 1 -- 172.21.15.117:0/3428890346 learned_addr learned my addr 172.21.15.117:0/3428890346 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:31.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922949131 0 0) 0x7f77b8141f30 con 0x7f77b80b2800 2024-03-29T10:35:31.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77b67fc640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.300+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a0003520 con 0x7f77b80b2800 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871112985 0 0) 0x7f77b80aad10 con 0x7f77b8142b20 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b8141f30 con 0x7f77b8142b20 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 618988240 0 0) 0x7f77a0003520 con 0x7f77b80b2800 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 >> v1:172.21.15.117:6790/0 conn(0x7f77b813c040 legacy=0x7f77b814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 >> v1:172.21.15.117:6789/0 conn(0x7f77b8142b20 legacy=0x7f77b813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 1 monclient: found mon.noname-c 2024-03-29T10:35:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b80b3900 con 0x7f77b80b2800 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient: _finish_auth 0 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77bcf42640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f77ac001690 con 0x7f77b80b2800 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77bcf42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77b67fc640 10 monclient: dump: 2024-03-29T10:35:31.289 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.301+0000 7f77bcf42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.302+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77ac0017f0 con 0x7f77b80b2800 2024-03-29T10:35:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.302+0000 7f77b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.302+0000 7f77b67fc640 1 -- 172.21.15.117:0/3428890346 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f77ac001c00 con 0x7f77b80b2800 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.302+0000 7f77b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.302+0000 7f77b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.302+0000 7f77b67fc640 10 monclient: dump: 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:31.291 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:31.292 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:31.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.302+0000 7f77bcf42640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:31.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.302+0000 7f77bcf42640 4 set_mon_vals no callback set 2024-03-29T10:35:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.303+0000 7f77bcf42640 10 monclient: shutdown 2024-03-29T10:35:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.303+0000 7f77bcf42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.303+0000 7f77bcf42640 1 -- 172.21.15.117:0/3428890346 >> v1:172.21.15.136:6789/0 conn(0x7f77b80b2800 legacy=0x7f77b8142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.303+0000 7f77bcf42640 1 -- 172.21.15.117:0/3428890346 shutdown_connections 2024-03-29T10:35:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.303+0000 7f77bcf42640 1 -- 172.21.15.117:0/3428890346 >> 172.21.15.117:0/3428890346 conn(0x7f77b806d0e0 msgr2=0x7f77b8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.303+0000 7f77bcf42640 1 -- 172.21.15.117:0/3428890346 shutdown_connections 2024-03-29T10:35:31.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.303+0000 7f77bcf42640 1 -- 172.21.15.117:0/3428890346 wait complete. 2024-03-29T10:35:31.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.304+0000 7f77bcf42640 10 monclient: build_initial_monmap 2024-03-29T10:35:31.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.304+0000 7f77bcf42640 10 monclient: monmap: 2024-03-29T10:35:31.299 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:31.299 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:31.299 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:35:31.299 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:35:31.299 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.305+0000 7f77bcf42640 1 librados: starting msgr at 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.305+0000 7f77bcf42640 1 librados: starting objecter 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.305+0000 7f77bcf42640 1 Processor -- start 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.305+0000 7f77bcf42640 1 -- start start 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.305+0000 7f77bcf42640 1 librados: setting wanted keys 2024-03-29T10:35:31.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.305+0000 7f77bcf42640 1 librados: calling monclient init 2024-03-29T10:35:31.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.305+0000 7f77bcf42640 10 monclient: init 2024-03-29T10:35:31.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.306+0000 7f77bcf42640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:31.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.306+0000 7f77bcf42640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:35:31.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.306+0000 7f77bcf42640 10 monclient(hunting): picked mon.noname-a con 0x7f77b813c040 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:31.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f77bcf42640 10 monclient(hunting): picked mon.noname-c con 0x7f77b8142b20 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:31.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f77bcf42640 10 monclient(hunting): picked mon.noname-b con 0x7f77b813a2c0 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:31.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f77bcf42640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b81dee90 con 0x7f77b813c040 2024-03-29T10:35:31.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f77bcf42640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b813a0b0 con 0x7f77b813a2c0 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f77bcf42640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b8071bd0 con 0x7f77b8142b20 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f77bcf42640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f77bcf42640 10 monclient(hunting): authenticate will time out at 852.569226s 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f77b77fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f77b813c040 0x7f77b81df320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:53288/0 (socket says 172.21.15.117:53288) 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f77b77fe640 1 -- 172.21.15.117:0/176725998 learned_addr learned my addr 172.21.15.117:0/176725998 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f779ffff640 10 client.?.objecter ms_handle_connect 0x7f77b813c040 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f779ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f779ffff640 10 client.?.objecter ms_handle_connect 0x7f77b8142b20 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f779ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:31.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f779ffff640 10 client.?.objecter ms_handle_connect 0x7f77b813a2c0 2024-03-29T10:35:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.307+0000 7f779ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688800128 0 0) 0x7f77b813a0b0 con 0x7f77b813a2c0 2024-03-29T10:35:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:35:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a4004eb0 con 0x7f77b813a2c0 2024-03-29T10:35:31.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294186797 0 0) 0x7f77b8071bd0 con 0x7f77b8142b20 2024-03-29T10:35:31.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:35:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b813a0b0 con 0x7f77b8142b20 2024-03-29T10:35:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.308+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1054326321 0 0) 0x7f77a4004eb0 con 0x7f77b813a2c0 2024-03-29T10:35:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77b8071bd0 con 0x7f77b813a2c0 2024-03-29T10:35:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f77a8001840 con 0x7f77b813a2c0 2024-03-29T10:35:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): dump: 2024-03-29T10:35:31.306 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:31.306 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973333906 0 0) 0x7f77b81dee90 con 0x7f77b813c040 2024-03-29T10:35:31.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:35:31.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a4004eb0 con 0x7f77b813c040 2024-03-29T10:35:31.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 52228628 0 0) 0x7f77b813a0b0 con 0x7f77b8142b20 2024-03-29T10:35:31.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77b81dee90 con 0x7f77b8142b20 2024-03-29T10:35:31.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f77b0001b90 con 0x7f77b8142b20 2024-03-29T10:35:31.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:31.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:31.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 10 monclient(hunting): dump: 2024-03-29T10:35:31.309 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:31.309 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:31.309 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.309 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.309 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:31.309 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:31.309 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:31.310 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:31.310 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:31.310 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.309+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 891289916 0 0) 0x7f77b8071bd0 con 0x7f77b813a2c0 2024-03-29T10:35:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 >> v1:172.21.15.117:6790/0 conn(0x7f77b8142b20 legacy=0x7f77b8139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 >> v1:172.21.15.117:6789/0 conn(0x7f77b813c040 legacy=0x7f77b81df320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:35:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 1 monclient: found mon.b 2024-03-29T10:35:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b81dea60 con 0x7f77b813a2c0 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 10 monclient: _finish_auth 0 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f77bcf42640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f77bcf42640 10 monclient: _renew_subs 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f77bcf42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b8145720 con 0x7f77b813a2c0 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f77bcf42640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f77bcf42640 10 monclient: _renew_subs 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f77bcf42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:35:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b806f850 con 0x7f77b813a2c0 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f77bcf42640 1 librados: init done 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77a8001d50 con 0x7f77b813a2c0 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.310+0000 7f779ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.311+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f77a8003790 con 0x7f77b813a2c0 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.312+0000 7f77bcf42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.312+0000 7f77bcf42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.312+0000 7f77bcf42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.312+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b804f730 con 0x7f77b813a2c0 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.312+0000 7f779ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.312+0000 7f779ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.312+0000 7f779ffff640 10 monclient: dump: 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:31.312 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.312+0000 7f77b5ffb640 4 set_mon_vals no callback set 2024-03-29T10:35:31.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.313+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f77a8019820 con 0x7f77b813a2c0 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.314+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f77a80837c0 con 0x7f77b813a2c0 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.314+0000 7f779ffff640 10 client.4270.objecter ms_dispatch 0x7f77b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.314+0000 7f779ffff640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.314+0000 7f779ffff640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.314+0000 7f779ffff640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.314+0000 7f77b77fe640 10 monclient: get_auth_request con 0x7f77a406ddd0 auth_method 0 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.315+0000 7f779ffff640 10 client.4270.objecter ms_handle_connect 0x7f77a406ddd0 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.320+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77a804eed0 con 0x7f77b813a2c0 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.321+0000 7f779ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.321+0000 7f779ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:31.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.710+0000 7f77bcf42640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:31.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.710+0000 7f77bcf42640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:31.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.710+0000 7f77bcf42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:35:31.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.710+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77b81debf0 con 0x7f77b813a2c0 2024-03-29T10:35:31.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.711+0000 7f779ffff640 1 -- 172.21.15.117:0/176725998 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f77a8050fd0 con 0x7f77b813a2c0 2024-03-29T10:35:31.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.712+0000 7f779ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:31.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.712+0000 7f779ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:31.697 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:35:31.697 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:31.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 10 librados: watch_flush enter 2024-03-29T10:35:31.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 10 librados: watch_flush exit 2024-03-29T10:35:31.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T10:35:31.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 10 client.4270.objecter successfully canceled tick 2024-03-29T10:35:31.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 >> v1:172.21.15.117:6816/376768171 conn(0x7f77a406ddd0 legacy=0x7f77a4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 10 monclient: shutdown 2024-03-29T10:35:31.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:31.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 >> v1:172.21.15.136:6789/0 conn(0x7f77b813a2c0 legacy=0x7f77b81399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 shutdown_connections 2024-03-29T10:35:31.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 >> 172.21.15.117:0/176725998 conn(0x7f77b806d0e0 msgr2=0x7f77b80b83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:31.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 shutdown_connections 2024-03-29T10:35:31.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 1 -- 172.21.15.117:0/176725998 wait complete. 2024-03-29T10:35:31.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:31.716+0000 7f77bcf42640 1 librados: shutdown 2024-03-29T10:35:31.715 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:38.523 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:38.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.672+0000 7ff193c93640 10 monclient: get_monmap_and_config 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.672+0000 7ff193c93640 10 monclient: build_initial_monmap 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.672+0000 7ff193c93640 10 monclient: monmap: 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:35:38.673809+0000 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:35:38.673809+0000 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:35:38.658 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:38.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.673+0000 7ff193c93640 1 Processor -- start 2024-03-29T10:35:38.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.673+0000 7ff193c93640 1 -- start start 2024-03-29T10:35:38.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.673+0000 7ff193c93640 10 monclient: init 2024-03-29T10:35:38.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.674+0000 7ff193c93640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:38.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.674+0000 7ff193c93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff193c93640 10 monclient(hunting): picked mon.noname-a con 0x7ff18c0b0770 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff193c93640 10 monclient(hunting): picked mon.noname-b con 0x7ff18c0ad850 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff193c93640 10 monclient(hunting): picked mon.noname-c con 0x7ff18c13c1b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff193c93640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0ac8e0 con 0x7ff18c0b0770 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff193c93640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0ad330 con 0x7ff18c13c1b0 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff193c93640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0acfe0 con 0x7ff18c0ad850 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff193c93640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff193c93640 10 monclient(hunting): authenticate will time out at 859.938974s 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff19122e640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ff18c0ad850 0x7ff18c0aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41386/0 (socket says 172.21.15.117:41386) 2024-03-29T10:35:38.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.676+0000 7ff19122e640 1 -- 172.21.15.117:0/3889408754 learned_addr learned my addr 172.21.15.117:0/3889408754 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:38.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.677+0000 7ff190a2d640 1 -- 172.21.15.117:0/3889408754 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333098609 0 0) 0x7ff18c0acfe0 con 0x7ff18c0ad850 2024-03-29T10:35:38.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.677+0000 7ff190a2d640 10 monclient(hunting): _negotiate 2024-03-29T10:35:38.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.677+0000 7ff190a2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:38.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.677+0000 7ff190a2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:38.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.677+0000 7ff190a2d640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:35:38.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.677+0000 7ff190a2d640 1 -- 172.21.15.117:0/3889408754 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff178003520 con 0x7ff18c0ad850 2024-03-29T10:35:38.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.677+0000 7ff190a2d640 1 -- 172.21.15.117:0/3889408754 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2492426933 0 0) 0x7ff178003520 con 0x7ff18c0ad850 2024-03-29T10:35:38.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.677+0000 7ff190a2d640 1 -- 172.21.15.117:0/3889408754 >> v1:172.21.15.136:6789/0 conn(0x7ff18c13c1b0 legacy=0x7ff18c14a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.677+0000 7ff190a2d640 1 -- 172.21.15.117:0/3889408754 >> v1:172.21.15.117:6789/0 conn(0x7ff18c0b0770 legacy=0x7ff18c072070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:38.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 10 monclient: _finish_hunting 0 2024-03-29T10:35:38.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 1 monclient: found mon.noname-b 2024-03-29T10:35:38.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:38.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:35:38.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 1 -- 172.21.15.117:0/3889408754 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0ab1f0 con 0x7ff18c0ad850 2024-03-29T10:35:38.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 10 monclient: _finish_auth 0 2024-03-29T10:35:38.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 10 monclient: _check_auth_tickets 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 1 -- 172.21.15.117:0/3889408754 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff17c001690 con 0x7ff18c0ad850 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff190a2d640 10 monclient: dump: 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:38.665 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:38.666 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:38.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff193c93640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:35:38.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.678+0000 7ff193c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:38.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.679+0000 7ff190a2d640 1 -- 172.21.15.117:0/3889408754 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff17c001920 con 0x7ff18c0ad850 2024-03-29T10:35:38.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.679+0000 7ff190a2d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:38.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.679+0000 7ff190a2d640 1 -- 172.21.15.117:0/3889408754 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff17c001d30 con 0x7ff18c0ad850 2024-03-29T10:35:38.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.679+0000 7ff190a2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:38.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.679+0000 7ff190a2d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:38.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.679+0000 7ff190a2d640 10 monclient: dump: 2024-03-29T10:35:38.667 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:38.667 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:38.667 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.667 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.667 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:38.667 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:38.668 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:38.668 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:38.668 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:38.668 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:38.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.679+0000 7ff193c93640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:38.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.679+0000 7ff193c93640 4 set_mon_vals no callback set 2024-03-29T10:35:38.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.680+0000 7ff193c93640 10 monclient: shutdown 2024-03-29T10:35:38.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.680+0000 7ff193c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:38.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.680+0000 7ff193c93640 1 -- 172.21.15.117:0/3889408754 >> v1:172.21.15.117:6790/0 conn(0x7ff18c0ad850 legacy=0x7ff18c0aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.680+0000 7ff193c93640 1 -- 172.21.15.117:0/3889408754 shutdown_connections 2024-03-29T10:35:38.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.680+0000 7ff193c93640 1 -- 172.21.15.117:0/3889408754 >> 172.21.15.117:0/3889408754 conn(0x7ff18c06b660 msgr2=0x7ff18c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:38.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.680+0000 7ff193c93640 1 -- 172.21.15.117:0/3889408754 shutdown_connections 2024-03-29T10:35:38.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.680+0000 7ff193c93640 1 -- 172.21.15.117:0/3889408754 wait complete. 2024-03-29T10:35:38.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.681+0000 7ff193c93640 10 monclient: build_initial_monmap 2024-03-29T10:35:38.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.681+0000 7ff193c93640 10 monclient: monmap: 2024-03-29T10:35:38.679 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:38.679 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:38.679 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:35:38.679 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:35:38.679 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:38.679 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:38.679 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:38.679 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:35:38.679 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:35:38.680 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:38.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.681+0000 7ff193c93640 1 librados: starting msgr at 2024-03-29T10:35:38.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.681+0000 7ff193c93640 1 librados: starting objecter 2024-03-29T10:35:38.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.681+0000 7ff193c93640 1 Processor -- start 2024-03-29T10:35:38.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.681+0000 7ff193c93640 1 -- start start 2024-03-29T10:35:38.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.681+0000 7ff193c93640 1 librados: setting wanted keys 2024-03-29T10:35:38.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.681+0000 7ff193c93640 1 librados: calling monclient init 2024-03-29T10:35:38.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.681+0000 7ff193c93640 10 monclient: init 2024-03-29T10:35:38.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.682+0000 7ff193c93640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:38.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.682+0000 7ff193c93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:35:38.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff193c93640 10 monclient(hunting): picked mon.noname-a con 0x7ff18c0ad850 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:38.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff193c93640 10 monclient(hunting): picked mon.noname-c con 0x7ff18c0b0770 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:38.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff193c93640 10 monclient(hunting): picked mon.noname-b con 0x7ff18c13c1b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:38.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff193c93640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c1c7c70 con 0x7ff18c0ad850 2024-03-29T10:35:38.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff193c93640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c1c7ec0 con 0x7ff18c13c1b0 2024-03-29T10:35:38.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff193c93640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c1c81a0 con 0x7ff18c0b0770 2024-03-29T10:35:38.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff193c93640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:38.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff193c93640 10 monclient(hunting): authenticate will time out at 859.945818s 2024-03-29T10:35:38.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff192230640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff18c13c1b0 0x7ff18c1cb760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:51926/0 (socket says 172.21.15.117:51926) 2024-03-29T10:35:38.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff192230640 1 -- 172.21.15.117:0/339945617 learned_addr learned my addr 172.21.15.117:0/339945617 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:38.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff1827fc640 10 client.?.objecter ms_handle_connect 0x7ff18c13c1b0 2024-03-29T10:35:38.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.683+0000 7ff1827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:38.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 client.?.objecter ms_handle_connect 0x7ff18c0b0770 2024-03-29T10:35:38.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:38.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268371364 0 0) 0x7ff18c1c81a0 con 0x7ff18c0b0770 2024-03-29T10:35:38.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:38.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:38.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:38.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:35:38.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff170004eb0 con 0x7ff18c0b0770 2024-03-29T10:35:38.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555369386 0 0) 0x7ff18c1c7ec0 con 0x7ff18c13c1b0 2024-03-29T10:35:38.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:38.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:38.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:38.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:35:38.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff18c1c81a0 con 0x7ff18c13c1b0 2024-03-29T10:35:38.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1285016734 0 0) 0x7ff18c1c81a0 con 0x7ff18c13c1b0 2024-03-29T10:35:38.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff18c1c7ec0 con 0x7ff18c13c1b0 2024-03-29T10:35:38.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff184001b90 con 0x7ff18c13c1b0 2024-03-29T10:35:38.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:38.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:38.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): dump: 2024-03-29T10:35:38.686 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:38.686 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:38.686 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.686 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.687 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:38.687 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:38.687 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:38.687 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:38.687 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:38.687 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 411643021 0 0) 0x7ff170004eb0 con 0x7ff18c0b0770 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff18c1c81a0 con 0x7ff18c0b0770 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff17c001650 con 0x7ff18c0b0770 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.684+0000 7ff1827fc640 10 monclient(hunting): dump: 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:38.688 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.685+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4008525946 0 0) 0x7ff18c1c81a0 con 0x7ff18c0b0770 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.685+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 >> v1:172.21.15.136:6789/0 conn(0x7ff18c13c1b0 legacy=0x7ff18c1cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.685+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 >> v1:172.21.15.117:6789/0 conn(0x7ff18c0ad850 legacy=0x7ff18c0b2750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff1827fc640 10 monclient: _finish_hunting 0 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff1827fc640 1 monclient: found mon.c 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff1827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff1827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0b2e60 con 0x7ff18c0b0770 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff1827fc640 10 monclient: _finish_auth 0 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff1827fc640 10 monclient: _check_auth_tickets 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff1827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:38.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff193c93640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff193c93640 10 monclient: _renew_subs 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff193c93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c0b3110 con 0x7ff18c0b0770 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff193c93640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff193c93640 10 monclient: _renew_subs 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff193c93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c1e7190 con 0x7ff18c0b0770 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.686+0000 7ff193c93640 1 librados: init done 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.687+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff17c0010a0 con 0x7ff18c0b0770 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.687+0000 7ff1827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.687+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff17c004dd0 con 0x7ff18c0b0770 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.687+0000 7ff1827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:38.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.687+0000 7ff1827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.687+0000 7ff183fff640 4 set_mon_vals no callback set 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.687+0000 7ff1827fc640 10 monclient: dump: 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.689+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7ff17c01ada0 con 0x7ff18c0b0770 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.689+0000 7ff193c93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.689+0000 7ff193c93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:38.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.689+0000 7ff193c93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:38.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.689+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff174005080 con 0x7ff18c0b0770 2024-03-29T10:35:38.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.689+0000 7ff191a2f640 10 monclient: get_auth_request con 0x7ff17006de70 auth_method 0 2024-03-29T10:35:38.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.690+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7ff17c085b70 con 0x7ff18c0b0770 2024-03-29T10:35:38.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.690+0000 7ff1827fc640 10 client.4292.objecter ms_dispatch 0x7ff18c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:38.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.690+0000 7ff1827fc640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:38.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.690+0000 7ff1827fc640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:38.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.690+0000 7ff1827fc640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T10:35:38.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.690+0000 7ff1827fc640 10 client.4292.objecter ms_handle_connect 0x7ff17006de70 2024-03-29T10:35:38.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.697+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff17c050420 con 0x7ff18c0b0770 2024-03-29T10:35:38.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.697+0000 7ff1827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:38.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:38.697+0000 7ff1827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:38.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.012+0000 7ff193c93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:38.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.012+0000 7ff193c93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:38.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.012+0000 7ff193c93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:38.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.012+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff174005330 con 0x7ff18c0b0770 2024-03-29T10:35:38.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.013+0000 7ff1827fc640 1 -- 172.21.15.117:0/339945617 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7ff17c001210 con 0x7ff18c0b0770 2024-03-29T10:35:38.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.013+0000 7ff1827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:38.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.013+0000 7ff1827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:38.999 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:35:38.999 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:39.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 10 librados: watch_flush enter 2024-03-29T10:35:39.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 10 librados: watch_flush exit 2024-03-29T10:35:39.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 10 client.4292.objecter successfully canceled tick 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 >> v1:172.21.15.117:6816/376768171 conn(0x7ff17006de70 legacy=0x7ff170070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 10 monclient: shutdown 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 >> v1:172.21.15.117:6790/0 conn(0x7ff18c0b0770 legacy=0x7ff18c1cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 shutdown_connections 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 >> 172.21.15.117:0/339945617 conn(0x7ff18c06b660 msgr2=0x7ff18c0aa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 shutdown_connections 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 1 -- 172.21.15.117:0/339945617 wait complete. 2024-03-29T10:35:39.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:39.017+0000 7ff193c93640 1 librados: shutdown 2024-03-29T10:35:39.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:45.826 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:45.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.969+0000 7fa53fd4b640 10 monclient: get_monmap_and_config 2024-03-29T10:35:45.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.969+0000 7fa53fd4b640 10 monclient: build_initial_monmap 2024-03-29T10:35:45.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.969+0000 7fa53fd4b640 10 monclient: monmap: 2024-03-29T10:35:45.955 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:45.955 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:45.956 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:35:45.971445+0000 2024-03-29T10:35:45.956 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:35:45.971445+0000 2024-03-29T10:35:45.956 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:45.956 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:45.956 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:45.956 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:35:45.956 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:35:45.956 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:45.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.970+0000 7fa53fd4b640 1 Processor -- start 2024-03-29T10:35:45.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.970+0000 7fa53fd4b640 1 -- start start 2024-03-29T10:35:45.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.970+0000 7fa53fd4b640 10 monclient: init 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 10 monclient(hunting): picked mon.noname-c con 0x7fa5381457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 10 monclient(hunting): picked mon.noname-b con 0x7fa53813f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 10 monclient(hunting): picked mon.noname-a con 0x7fa5380a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5380aaef0 con 0x7fa5380a9d70 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa538145bd0 con 0x7fa5381457f0 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa538145eb0 con 0x7fa53813f650 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.971+0000 7fa53fd4b640 10 monclient(hunting): authenticate will time out at 867.234864s 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.972+0000 7fa53d2e6640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fa53813f650 0x7fa5380b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44472/0 (socket says 172.21.15.117:44472) 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.972+0000 7fa53d2e6640 1 -- 172.21.15.117:0/2542668626 learned_addr learned my addr 172.21.15.117:0/2542668626 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:45.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.972+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515293559 0 0) 0x7fa538145eb0 con 0x7fa53813f650 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.972+0000 7fa53cae5640 10 monclient(hunting): _negotiate 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.972+0000 7fa53cae5640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.972+0000 7fa53cae5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.972+0000 7fa53cae5640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.972+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa524003520 con 0x7fa53813f650 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.973+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951088953 0 0) 0x7fa538145bd0 con 0x7fa5381457f0 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.973+0000 7fa53cae5640 10 monclient(hunting): _negotiate 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.973+0000 7fa53cae5640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.973+0000 7fa53cae5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.973+0000 7fa53cae5640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.973+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa538145eb0 con 0x7fa5381457f0 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.973+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1811068680 0 0) 0x7fa524003520 con 0x7fa53813f650 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.973+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 >> v1:172.21.15.136:6789/0 conn(0x7fa5381457f0 legacy=0x7fa53813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:45.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.973+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 >> v1:172.21.15.117:6789/0 conn(0x7fa5380a9d70 legacy=0x7fa53814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.974+0000 7fa53cae5640 10 monclient: _finish_hunting 0 2024-03-29T10:35:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.974+0000 7fa53cae5640 1 monclient: found mon.noname-b 2024-03-29T10:35:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:35:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538146340 con 0x7fa53813f650 2024-03-29T10:35:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: _finish_auth 0 2024-03-29T10:35:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: _check_auth_tickets 2024-03-29T10:35:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fa534001a10 con 0x7fa53813f650 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: dump: 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa534001c30 con 0x7fa53813f650 2024-03-29T10:35:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 1 -- 172.21.15.117:0/2542668626 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fa534001fc0 con 0x7fa53813f650 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53cae5640 10 monclient: dump: 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53fd4b640 5 monclient: authenticate success, global_id 4295 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53fd4b640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.975+0000 7fa53fd4b640 4 set_mon_vals no callback set 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.976+0000 7fa53fd4b640 10 monclient: shutdown 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.976+0000 7fa53fd4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.976+0000 7fa53fd4b640 1 -- 172.21.15.117:0/2542668626 >> v1:172.21.15.117:6790/0 conn(0x7fa53813f650 legacy=0x7fa5380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.976+0000 7fa53fd4b640 1 -- 172.21.15.117:0/2542668626 shutdown_connections 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.976+0000 7fa53fd4b640 1 -- 172.21.15.117:0/2542668626 >> 172.21.15.117:0/2542668626 conn(0x7fa538066a00 msgr2=0x7fa538066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.976+0000 7fa53fd4b640 1 -- 172.21.15.117:0/2542668626 shutdown_connections 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.976+0000 7fa53fd4b640 1 -- 172.21.15.117:0/2542668626 wait complete. 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.977+0000 7fa53fd4b640 10 monclient: build_initial_monmap 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.977+0000 7fa53fd4b640 10 monclient: monmap: 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:45.969 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.977+0000 7fa53fd4b640 1 librados: starting msgr at 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.977+0000 7fa53fd4b640 1 librados: starting objecter 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 1 Processor -- start 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 1 -- start start 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 1 librados: setting wanted keys 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 1 librados: calling monclient init 2024-03-29T10:35:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 10 monclient: init 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 10 monclient(hunting): picked mon.noname-c con 0x7fa5380a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 10 monclient(hunting): picked mon.noname-a con 0x7fa53813f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 10 monclient(hunting): picked mon.noname-b con 0x7fa5381457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53dae7640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fa5380a9d70 0x7fa53813b420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44736/0 (socket says 172.21.15.117:44736) 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5381cb860 con 0x7fa53813f650 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5381ccc80 con 0x7fa5381457f0 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5381ccf60 con 0x7fa5380a9d70 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53fd4b640 10 monclient(hunting): authenticate will time out at 867.242040s 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.978+0000 7fa53dae7640 1 -- 172.21.15.117:0/3620380348 learned_addr learned my addr 172.21.15.117:0/3620380348 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 client.?.objecter ms_handle_connect 0x7fa5380a9d70 2024-03-29T10:35:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806252179 0 0) 0x7fa5381ccf60 con 0x7fa5380a9d70 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c004eb0 con 0x7fa5380a9d70 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1318369597 0 0) 0x7fa51c004eb0 con 0x7fa5380a9d70 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5381ccf60 con 0x7fa5380a9d70 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fa528001970 con 0x7fa5380a9d70 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 monclient(hunting): dump: 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:45.972 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.979+0000 7fa52e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.980+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3391078128 0 0) 0x7fa5381ccf60 con 0x7fa5380a9d70 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.980+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 >> v1:172.21.15.136:6789/0 conn(0x7fa5381457f0 legacy=0x7fa5381ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.980+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 >> v1:172.21.15.117:6789/0 conn(0x7fa53813f650 legacy=0x7fa538138a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:35:45.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa52e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:35:45.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa52e7fc640 1 monclient: found mon.c 2024-03-29T10:35:45.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa52e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:45.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa52e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:45.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5381cb340 con 0x7fa5380a9d70 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa52e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa52e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa52e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa53fd4b640 5 monclient: authenticate success, global_id 4298 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa53fd4b640 10 monclient: _renew_subs 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa53fd4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5381cb530 con 0x7fa5380a9d70 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa53fd4b640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa53fd4b640 10 monclient: _renew_subs 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa53fd4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5381cbed0 con 0x7fa5380a9d70 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.982+0000 7fa53fd4b640 1 librados: init done 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.983+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa528001d50 con 0x7fa5380a9d70 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.983+0000 7fa52e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:45.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.983+0000 7fa52ffff640 4 set_mon_vals no callback set 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.983+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fa528001bd0 con 0x7fa5380a9d70 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.983+0000 7fa52e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.983+0000 7fa52e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.983+0000 7fa52e7fc640 10 monclient: dump: 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:45.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.984+0000 7fa53fd4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.984+0000 7fa53fd4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.984+0000 7fa53fd4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.984+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa520005080 con 0x7fa5380a9d70 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.985+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fa528003a60 con 0x7fa5380a9d70 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.986+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fa528084b00 con 0x7fa5380a9d70 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.986+0000 7fa52e7fc640 10 client.4298.objecter ms_dispatch 0x7fa538000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.986+0000 7fa52e7fc640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.986+0000 7fa52e7fc640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.986+0000 7fa52e7fc640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.986+0000 7fa53d2e6640 10 monclient: get_auth_request con 0x7fa51c076420 auth_method 0 2024-03-29T10:35:45.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.987+0000 7fa52e7fc640 10 client.4298.objecter ms_handle_connect 0x7fa51c076420 2024-03-29T10:35:45.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.993+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5280503e0 con 0x7fa5380a9d70 2024-03-29T10:35:45.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.993+0000 7fa52e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:45.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:45.993+0000 7fa52e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:46.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.312+0000 7fa53fd4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:46.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.312+0000 7fa53fd4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:46.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.312+0000 7fa53fd4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:35:46.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.312+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa520005330 con 0x7fa5380a9d70 2024-03-29T10:35:46.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.313+0000 7fa52e7fc640 1 -- 172.21.15.117:0/3620380348 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fa528052310 con 0x7fa5380a9d70 2024-03-29T10:35:46.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.313+0000 7fa52e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:46.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.313+0000 7fa52e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:46.300 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:35:46.300 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 10 librados: watch_flush enter 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 10 librados: watch_flush exit 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 10 client.4298.objecter successfully canceled tick 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 >> v1:172.21.15.117:6816/376768171 conn(0x7fa51c076420 legacy=0x7fa51c0788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 10 monclient: shutdown 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 >> v1:172.21.15.117:6790/0 conn(0x7fa5380a9d70 legacy=0x7fa53813b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 shutdown_connections 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.316+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 >> 172.21.15.117:0/3620380348 conn(0x7fa538066a00 msgr2=0x7fa5380b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.317+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 shutdown_connections 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.317+0000 7fa53fd4b640 1 -- 172.21.15.117:0/3620380348 wait complete. 2024-03-29T10:35:46.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:46.317+0000 7fa53fd4b640 1 librados: shutdown 2024-03-29T10:35:46.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:53.123 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.286+0000 7fefeec63640 10 monclient: get_monmap_and_config 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.286+0000 7fefeec63640 10 monclient: build_initial_monmap 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.286+0000 7fefeec63640 10 monclient: monmap: 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:35:53.287822+0000 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:35:53.287822+0000 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:35:53.272 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:35:53.273 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:53.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.287+0000 7fefeec63640 1 Processor -- start 2024-03-29T10:35:53.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.288+0000 7fefeec63640 1 -- start start 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.288+0000 7fefeec63640 10 monclient: init 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.288+0000 7fefeec63640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.288+0000 7fefeec63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.288+0000 7fefeec63640 10 monclient(hunting): picked mon.noname-b con 0x7fefe8071f40 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.288+0000 7fefeec63640 10 monclient(hunting): picked mon.noname-c con 0x7fefe813f7e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.289+0000 7fefeec63640 10 monclient(hunting): picked mon.noname-a con 0x7fefe8142750 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.289+0000 7fefeec63640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe80aef80 con 0x7fefe8142750 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.289+0000 7fefeec63640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe80affc0 con 0x7fefe813f7e0 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.289+0000 7fefeec63640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe80aad20 con 0x7fefe8071f40 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.289+0000 7fefeec63640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:53.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.289+0000 7fefeec63640 10 monclient(hunting): authenticate will time out at 874.551242s 2024-03-29T10:35:53.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.289+0000 7fefed460640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fefe813f7e0 0x7fefe8142040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:58180/0 (socket says 172.21.15.117:58180) 2024-03-29T10:35:53.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.289+0000 7fefed460640 1 -- 172.21.15.117:0/2499531319 learned_addr learned my addr 172.21.15.117:0/2499531319 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:53.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.290+0000 7fefecc5f640 1 -- 172.21.15.117:0/2499531319 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329659763 0 0) 0x7fefe80affc0 con 0x7fefe813f7e0 2024-03-29T10:35:53.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.290+0000 7fefecc5f640 10 monclient(hunting): _negotiate 2024-03-29T10:35:53.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.290+0000 7fefecc5f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:53.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.290+0000 7fefecc5f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:53.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.290+0000 7fefecc5f640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:35:53.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.290+0000 7fefecc5f640 1 -- 172.21.15.117:0/2499531319 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd4003520 con 0x7fefe813f7e0 2024-03-29T10:35:53.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.291+0000 7fefecc5f640 1 -- 172.21.15.117:0/2499531319 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 627436002 0 0) 0x7fefd4003520 con 0x7fefe813f7e0 2024-03-29T10:35:53.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.291+0000 7fefecc5f640 1 -- 172.21.15.117:0/2499531319 >> v1:172.21.15.117:6790/0 conn(0x7fefe8071f40 legacy=0x7fefe813f0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:53.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.291+0000 7fefecc5f640 1 -- 172.21.15.117:0/2499531319 >> v1:172.21.15.117:6789/0 conn(0x7fefe8142750 legacy=0x7fefe814d960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:53.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 10 monclient: _finish_hunting 0 2024-03-29T10:35:53.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 1 monclient: found mon.noname-c 2024-03-29T10:35:53.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:53.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:35:53.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 1 -- 172.21.15.117:0/2499531319 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe80ab000 con 0x7fefe813f7e0 2024-03-29T10:35:53.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 10 monclient: _finish_auth 0 2024-03-29T10:35:53.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 10 monclient: _check_auth_tickets 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 1 -- 172.21.15.117:0/2499531319 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fefd8001690 con 0x7fefe813f7e0 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefecc5f640 10 monclient: dump: 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:53.278 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:53.279 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:53.279 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:53.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefeec63640 5 monclient: authenticate success, global_id 4282 2024-03-29T10:35:53.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.292+0000 7fefeec63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:53.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.293+0000 7fefecc5f640 1 -- 172.21.15.117:0/2499531319 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefd80017f0 con 0x7fefe813f7e0 2024-03-29T10:35:53.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.293+0000 7fefecc5f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.293+0000 7fefecc5f640 1 -- 172.21.15.117:0/2499531319 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fefd8001c40 con 0x7fefe813f7e0 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.293+0000 7fefecc5f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.293+0000 7fefecc5f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.293+0000 7fefecc5f640 10 monclient: dump: 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:53.280 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:53.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.293+0000 7fefeec63640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:53.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.293+0000 7fefeec63640 4 set_mon_vals no callback set 2024-03-29T10:35:53.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.294+0000 7fefeec63640 10 monclient: shutdown 2024-03-29T10:35:53.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.294+0000 7fefeec63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:53.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.294+0000 7fefeec63640 1 -- 172.21.15.117:0/2499531319 >> v1:172.21.15.136:6789/0 conn(0x7fefe813f7e0 legacy=0x7fefe8142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.294+0000 7fefeec63640 1 -- 172.21.15.117:0/2499531319 shutdown_connections 2024-03-29T10:35:53.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.294+0000 7fefeec63640 1 -- 172.21.15.117:0/2499531319 >> 172.21.15.117:0/2499531319 conn(0x7fefe80673e0 msgr2=0x7fefe80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:53.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.296+0000 7fefeec63640 1 -- 172.21.15.117:0/2499531319 shutdown_connections 2024-03-29T10:35:53.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.296+0000 7fefeec63640 1 -- 172.21.15.117:0/2499531319 wait complete. 2024-03-29T10:35:53.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.297+0000 7fefeec63640 10 monclient: build_initial_monmap 2024-03-29T10:35:53.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.298+0000 7fefeec63640 10 monclient: monmap: 2024-03-29T10:35:53.282 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:35:53.282 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:53.282 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:35:53.282 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:35:53.282 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:53.282 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:53.283 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:35:53.283 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:35:53.283 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:35:53.283 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:53.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.298+0000 7fefeec63640 1 librados: starting msgr at 2024-03-29T10:35:53.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.298+0000 7fefeec63640 1 librados: starting objecter 2024-03-29T10:35:53.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.298+0000 7fefeec63640 1 Processor -- start 2024-03-29T10:35:53.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.298+0000 7fefeec63640 1 -- start start 2024-03-29T10:35:53.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.298+0000 7fefeec63640 1 librados: setting wanted keys 2024-03-29T10:35:53.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.298+0000 7fefeec63640 1 librados: calling monclient init 2024-03-29T10:35:53.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.298+0000 7fefeec63640 10 monclient: init 2024-03-29T10:35:53.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.299+0000 7fefeec63640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:53.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.299+0000 7fefeec63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:35:53.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.299+0000 7fefeec63640 10 monclient(hunting): picked mon.noname-b con 0x7fefe8071f40 addr v1:172.21.15.136:6789/0 2024-03-29T10:35:53.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.300+0000 7fefeec63640 10 monclient(hunting): picked mon.noname-a con 0x7fefe813f7e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:35:53.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.302+0000 7fefeec63640 10 monclient(hunting): picked mon.noname-c con 0x7fefe8142750 addr v1:172.21.15.117:6790/0 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.302+0000 7fefed460640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fefe813f7e0 0x7fefe81dad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:42017/0 (socket says 172.21.15.117:42017) 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.302+0000 7fefed460640 1 -- 172.21.15.117:0/2782046914 learned_addr learned my addr 172.21.15.117:0/2782046914 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.302+0000 7fefde7fc640 10 client.?.objecter ms_handle_connect 0x7fefe813f7e0 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.302+0000 7fefde7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe81da900 con 0x7fefe813f7e0 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe8143c00 con 0x7fefe8071f40 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe80acf40 con 0x7fefe8142750 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783212556 0 0) 0x7fefe81da900 con 0x7fefe813f7e0 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefeec63640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefeec63640 10 monclient(hunting): authenticate will time out at 874.565510s 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:53.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefcc004eb0 con 0x7fefe813f7e0 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 client.?.objecter ms_handle_connect 0x7fefe8142750 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797681927 0 0) 0x7fefe80acf40 con 0x7fefe8142750 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe81da900 con 0x7fefe8142750 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2585336058 0 0) 0x7fefcc004eb0 con 0x7fefe813f7e0 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.303+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefe80acf40 con 0x7fefe813f7e0 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fefd8001bc0 con 0x7fefe813f7e0 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:53.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient(hunting): dump: 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1111346576 0 0) 0x7fefe81da900 con 0x7fefe8142750 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefcc004eb0 con 0x7fefe8142750 2024-03-29T10:35:53.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fefe0001a90 con 0x7fefe8142750 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient(hunting): dump: 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2663400532 0 0) 0x7fefe80acf40 con 0x7fefe813f7e0 2024-03-29T10:35:53.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 >> v1:172.21.15.117:6790/0 conn(0x7fefe8142750 legacy=0x7fefe81e1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 >> v1:172.21.15.136:6789/0 conn(0x7fefe8071f40 legacy=0x7fefe80ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 monclient: found mon.a 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe81da5d0 con 0x7fefe813f7e0 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient: _finish_auth 0 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefeec63640 5 monclient: authenticate success, global_id 4284 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefeec63640 10 monclient: _renew_subs 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefeec63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe81db4a0 con 0x7fefe813f7e0 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefeec63640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefeec63640 10 monclient: _renew_subs 2024-03-29T10:35:53.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefeec63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe80abe60 con 0x7fefe813f7e0 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefd8004bf0 con 0x7fefe813f7e0 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fefd8001d80 con 0x7fefe813f7e0 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefde7fc640 10 monclient: dump: 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:35:53.298 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefeec63640 1 librados: init done 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.304+0000 7fefdffff640 4 set_mon_vals no callback set 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.306+0000 7fefeec63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.306+0000 7fefeec63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.306+0000 7fefeec63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.306+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe804f730 con 0x7fefe813f7e0 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.307+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fefd801ad70 con 0x7fefe813f7e0 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.307+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fefd8084b70 con 0x7fefe813f7e0 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.307+0000 7fefde7fc640 10 client.4284.objecter ms_dispatch 0x7fefe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.307+0000 7fefde7fc640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.307+0000 7fefde7fc640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.307+0000 7fefde7fc640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-29T10:35:53.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.307+0000 7fefedc61640 10 monclient: get_auth_request con 0x7fefcc06ddb0 auth_method 0 2024-03-29T10:35:53.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.308+0000 7fefde7fc640 10 client.4284.objecter ms_handle_connect 0x7fefcc06ddb0 2024-03-29T10:35:53.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.315+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefd8050450 con 0x7fefe813f7e0 2024-03-29T10:35:53.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.315+0000 7fefde7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:53.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.315+0000 7fefde7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:53.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.652+0000 7fefeec63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:53.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.652+0000 7fefeec63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:53.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.652+0000 7fefeec63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:35:53.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.652+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefe806a3e0 con 0x7fefe813f7e0 2024-03-29T10:35:53.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.653+0000 7fefde7fc640 1 -- 172.21.15.117:0/2782046914 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fefd8052380 con 0x7fefe813f7e0 2024-03-29T10:35:53.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.653+0000 7fefde7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:53.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.653+0000 7fefde7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:53.640 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:35:53.640 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:53.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.657+0000 7fefeec63640 10 librados: watch_flush enter 2024-03-29T10:35:53.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.657+0000 7fefeec63640 10 librados: watch_flush exit 2024-03-29T10:35:53.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.657+0000 7fefeec63640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-29T10:35:53.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.657+0000 7fefeec63640 10 client.4284.objecter successfully canceled tick 2024-03-29T10:35:53.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.657+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 >> v1:172.21.15.117:6816/376768171 conn(0x7fefcc06ddb0 legacy=0x7fefcc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.657+0000 7fefeec63640 10 monclient: shutdown 2024-03-29T10:35:53.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.658+0000 7fefeec63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:53.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.658+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 >> v1:172.21.15.117:6789/0 conn(0x7fefe813f7e0 legacy=0x7fefe81dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.658+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 shutdown_connections 2024-03-29T10:35:53.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.658+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 >> 172.21.15.117:0/2782046914 conn(0x7fefe80673e0 msgr2=0x7fefe8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:53.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.660+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 shutdown_connections 2024-03-29T10:35:53.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.661+0000 7fefeec63640 1 -- 172.21.15.117:0/2782046914 wait complete. 2024-03-29T10:35:53.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:35:53.661+0000 7fefeec63640 1 librados: shutdown 2024-03-29T10:35:53.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:00.468 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:00.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.627+0000 7f694f038640 10 monclient: get_monmap_and_config 2024-03-29T10:36:00.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.627+0000 7f694f038640 10 monclient: build_initial_monmap 2024-03-29T10:36:00.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.627+0000 7f694f038640 10 monclient: monmap: 2024-03-29T10:36:00.613 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:00.613 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:00.613 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:36:00.628239+0000 2024-03-29T10:36:00.613 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:36:00.628239+0000 2024-03-29T10:36:00.613 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:00.613 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:00.614 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:00.614 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:36:00.614 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:36:00.614 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:00.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.630+0000 7f694f038640 1 Processor -- start 2024-03-29T10:36:00.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.630+0000 7f694f038640 1 -- start start 2024-03-29T10:36:00.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.630+0000 7f694f038640 10 monclient: init 2024-03-29T10:36:00.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:00.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:00.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 10 monclient(hunting): picked mon.noname-b con 0x7f69480b4100 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:00.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 10 monclient(hunting): picked mon.noname-c con 0x7f694813ee90 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:00.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 10 monclient(hunting): picked mon.noname-a con 0x7f69480a9ac0 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:00.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69480af6f0 con 0x7f69480a9ac0 2024-03-29T10:36:00.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69480aac80 con 0x7f694813ee90 2024-03-29T10:36:00.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69480aaf60 con 0x7f69480b4100 2024-03-29T10:36:00.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:00.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.631+0000 7f694f038640 10 monclient(hunting): authenticate will time out at 881.893960s 2024-03-29T10:36:00.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.637+0000 7f6947fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f694813ee90 0x7f694813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:57929/0 (socket says 172.21.15.117:57929) 2024-03-29T10:36:00.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.637+0000 7f6947fff640 1 -- 172.21.15.117:0/2822283632 learned_addr learned my addr 172.21.15.117:0/2822283632 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:00.626 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702982470 0 0) 0x7f69480aaf60 con 0x7f69480b4100 2024-03-29T10:36:00.626 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:00.626 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:00.626 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:00.626 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:36:00.626 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6930003520 con 0x7f69480b4100 2024-03-29T10:36:00.626 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342904465 0 0) 0x7f69480af6f0 con 0x7f69480a9ac0 2024-03-29T10:36:00.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:00.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:00.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:00.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:36:00.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.640+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69480aaf60 con 0x7f69480a9ac0 2024-03-29T10:36:00.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641882827 0 0) 0x7f69480aac80 con 0x7f694813ee90 2024-03-29T10:36:00.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69480af6f0 con 0x7f694813ee90 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850731225 0 0) 0x7f69480aaf60 con 0x7f69480a9ac0 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 >> v1:172.21.15.117:6790/0 conn(0x7f69480b4100 legacy=0x7f69480adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 >> v1:172.21.15.136:6789/0 conn(0x7f694813ee90 legacy=0x7f694813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 1 monclient: found mon.noname-a 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694813e970 con 0x7f69480a9ac0 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:00.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f693c0015f0 con 0x7f69480a9ac0 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.642+0000 7f69477fe640 10 monclient: dump: 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:00.629 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:00.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.643+0000 7f694f038640 5 monclient: authenticate success, global_id 4287 2024-03-29T10:36:00.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.643+0000 7f694f038640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:00.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.644+0000 7f69477fe640 1 -- 172.21.15.117:0/2822283632 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f693c001750 con 0x7f69480a9ac0 2024-03-29T10:36:00.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.644+0000 7f69477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:00.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.645+0000 7f694f038640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:00.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.645+0000 7f694f038640 4 set_mon_vals no callback set 2024-03-29T10:36:00.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.645+0000 7f694f038640 10 monclient: shutdown 2024-03-29T10:36:00.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.645+0000 7f694f038640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:00.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.645+0000 7f694f038640 1 -- 172.21.15.117:0/2822283632 >> v1:172.21.15.117:6789/0 conn(0x7f69480a9ac0 legacy=0x7f6948142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.646+0000 7f694f038640 1 -- 172.21.15.117:0/2822283632 shutdown_connections 2024-03-29T10:36:00.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.646+0000 7f694f038640 1 -- 172.21.15.117:0/2822283632 >> 172.21.15.117:0/2822283632 conn(0x7f69480b35e0 msgr2=0x7f69480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:00.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.646+0000 7f694f038640 1 -- 172.21.15.117:0/2822283632 shutdown_connections 2024-03-29T10:36:00.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.646+0000 7f694f038640 1 -- 172.21.15.117:0/2822283632 wait complete. 2024-03-29T10:36:00.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.647+0000 7f694f038640 10 monclient: build_initial_monmap 2024-03-29T10:36:00.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.647+0000 7f694f038640 10 monclient: monmap: 2024-03-29T10:36:00.635 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:00.635 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.648+0000 7f694f038640 1 librados: starting msgr at 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.648+0000 7f694f038640 1 librados: starting objecter 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.648+0000 7f694f038640 1 Processor -- start 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.648+0000 7f694f038640 1 -- start start 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.648+0000 7f694f038640 1 librados: setting wanted keys 2024-03-29T10:36:00.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.648+0000 7f694f038640 1 librados: calling monclient init 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.648+0000 7f694f038640 10 monclient: init 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 10 monclient(hunting): picked mon.noname-c con 0x7f69480a9ac0 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 10 monclient(hunting): picked mon.noname-b con 0x7f69480b4100 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 10 monclient(hunting): picked mon.noname-a con 0x7f694813ee90 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694812abb0 con 0x7f694813ee90 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694806f170 con 0x7f69480b4100 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694806f450 con 0x7f69480a9ac0 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:00.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.649+0000 7f694f038640 10 monclient(hunting): authenticate will time out at 881.911617s 2024-03-29T10:36:00.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.653+0000 7f694d5d5640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f694813ee90 0x7f6948141f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:46623/0 (socket says 172.21.15.117:46623) 2024-03-29T10:36:00.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.653+0000 7f694d5d5640 1 -- 172.21.15.117:0/1656719307 learned_addr learned my addr 172.21.15.117:0/1656719307 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:00.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.653+0000 7f69457fa640 10 client.?.objecter ms_handle_connect 0x7f694813ee90 2024-03-29T10:36:00.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.653+0000 7f69457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.655+0000 7f69457fa640 10 client.?.objecter ms_handle_connect 0x7f69480a9ac0 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.655+0000 7f69457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.655+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530164092 0 0) 0x7f694806f450 con 0x7f69480a9ac0 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.655+0000 7f69457fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.655+0000 7f69457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.655+0000 7f69457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.655+0000 7f69457fa640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.655+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6934004eb0 con 0x7f69480a9ac0 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.655+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1832358207 0 0) 0x7f6934004eb0 con 0x7f69480a9ac0 2024-03-29T10:36:00.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.656+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f694806f450 con 0x7f69480a9ac0 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.656+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6940001cd0 con 0x7f69480a9ac0 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.656+0000 7f69457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.656+0000 7f69457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.656+0000 7f69457fa640 10 monclient(hunting): dump: 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:00.646 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.656+0000 7f69457fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.656+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 814963758 0 0) 0x7f694806f450 con 0x7f69480a9ac0 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.656+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 >> v1:172.21.15.136:6789/0 conn(0x7f69480b4100 legacy=0x7f694812b8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.656+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 >> v1:172.21.15.117:6789/0 conn(0x7f694813ee90 legacy=0x7f6948141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f69457fa640 10 monclient: _finish_hunting 0 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f69457fa640 1 monclient: found mon.c 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f69457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f69457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694806f730 con 0x7f69480a9ac0 2024-03-29T10:36:00.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f69457fa640 10 monclient: _finish_auth 0 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f69457fa640 10 monclient: _check_auth_tickets 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f69457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f694f038640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f694f038640 10 monclient: _renew_subs 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f694f038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694812bfd0 con 0x7f69480a9ac0 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f694f038640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f694f038640 10 monclient: _renew_subs 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f694f038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69481e7410 con 0x7f69480a9ac0 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f694f038640 1 librados: init done 2024-03-29T10:36:00.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.657+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6940003990 con 0x7f69480a9ac0 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.658+0000 7f69457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.658+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6940003d60 con 0x7f69480a9ac0 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.658+0000 7f6946ffd640 4 set_mon_vals no callback set 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.659+0000 7f694f038640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.659+0000 7f694f038640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.659+0000 7f694f038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.659+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6928005080 con 0x7f69480a9ac0 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.659+0000 7f69457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.659+0000 7f69457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.659+0000 7f69457fa640 10 monclient: dump: 2024-03-29T10:36:00.649 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:00.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.660+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f6940003ec0 con 0x7f69480a9ac0 2024-03-29T10:36:00.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.661+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f6940083c80 con 0x7f69480a9ac0 2024-03-29T10:36:00.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.661+0000 7f69457fa640 10 client.4307.objecter ms_dispatch 0x7f6948000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:00.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.661+0000 7f69457fa640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:00.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.661+0000 7f69457fa640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:00.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.661+0000 7f69457fa640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-29T10:36:00.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.661+0000 7f6947fff640 10 monclient: get_auth_request con 0x7f693406dd50 auth_method 0 2024-03-29T10:36:00.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.662+0000 7f69457fa640 10 client.4307.objecter ms_handle_connect 0x7f693406dd50 2024-03-29T10:36:00.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.674+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f694004f560 con 0x7f69480a9ac0 2024-03-29T10:36:00.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.674+0000 7f69457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:00.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.674+0000 7f69457fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:00.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.994+0000 7f694f038640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:00.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.994+0000 7f694f038640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:00.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.994+0000 7f694f038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:00.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.994+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6928005330 con 0x7f69480a9ac0 2024-03-29T10:36:00.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.997+0000 7f69457fa640 1 -- 172.21.15.117:0/1656719307 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f6940051490 con 0x7f69480a9ac0 2024-03-29T10:36:00.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.997+0000 7f69457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:00.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:00.997+0000 7f69457fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:00.984 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:36:00.984 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:00.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.003+0000 7f694f038640 10 librados: watch_flush enter 2024-03-29T10:36:00.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.003+0000 7f694f038640 10 librados: watch_flush exit 2024-03-29T10:36:00.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.003+0000 7f694f038640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-29T10:36:00.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.003+0000 7f694f038640 10 client.4307.objecter successfully canceled tick 2024-03-29T10:36:00.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.003+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 >> v1:172.21.15.117:6816/376768171 conn(0x7f693406dd50 legacy=0x7f6934070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.003+0000 7f694f038640 10 monclient: shutdown 2024-03-29T10:36:00.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.003+0000 7f694f038640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:00.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.003+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 >> v1:172.21.15.117:6790/0 conn(0x7f69480a9ac0 legacy=0x7f694806fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.004+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 shutdown_connections 2024-03-29T10:36:00.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.004+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 >> 172.21.15.117:0/1656719307 conn(0x7f69480b35e0 msgr2=0x7f69480b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:00.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.005+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 shutdown_connections 2024-03-29T10:36:00.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.005+0000 7f694f038640 1 -- 172.21.15.117:0/1656719307 wait complete. 2024-03-29T10:36:00.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:01.005+0000 7f694f038640 1 librados: shutdown 2024-03-29T10:36:01.009 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:07.816 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:07.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.983+0000 7f4139a4a640 10 monclient: get_monmap_and_config 2024-03-29T10:36:07.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.983+0000 7f4139a4a640 10 monclient: build_initial_monmap 2024-03-29T10:36:07.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.983+0000 7f4139a4a640 10 monclient: monmap: 2024-03-29T10:36:07.968 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:07.968 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:07.968 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:36:07.984370+0000 2024-03-29T10:36:07.969 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:36:07.984370+0000 2024-03-29T10:36:07.969 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:07.969 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:07.969 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:07.969 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:36:07.969 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:36:07.969 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:07.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.986+0000 7f4139a4a640 1 Processor -- start 2024-03-29T10:36:07.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.986+0000 7f4139a4a640 1 -- start start 2024-03-29T10:36:07.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.987+0000 7f4139a4a640 10 monclient: init 2024-03-29T10:36:07.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:07.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:07.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f41341457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:07.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f413413f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:07.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f41340a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:07.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41340aaef0 con 0x7f41340a9d70 2024-03-29T10:36:07.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4134145bd0 con 0x7f41341457f0 2024-03-29T10:36:07.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4134145eb0 con 0x7f413413f650 2024-03-29T10:36:07.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:07.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.988+0000 7f4139a4a640 10 monclient(hunting): authenticate will time out at 889.250346s 2024-03-29T10:36:07.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.991+0000 7f41337fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f41340a9d70 0x7f413414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:34413/0 (socket says 172.21.15.117:34413) 2024-03-29T10:36:07.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.991+0000 7f41337fe640 1 -- 172.21.15.117:0/3899181195 learned_addr learned my addr 172.21.15.117:0/3899181195 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:07.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.994+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861708320 0 0) 0x7f41340aaef0 con 0x7f41340a9d70 2024-03-29T10:36:07.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.994+0000 7f4131ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:07.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.994+0000 7f4131ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:07.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.994+0000 7f4131ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:07.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.994+0000 7f4131ffb640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:36:07.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.994+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c003520 con 0x7f41340a9d70 2024-03-29T10:36:07.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454638065 0 0) 0x7f4134145eb0 con 0x7f413413f650 2024-03-29T10:36:07.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:07.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:07.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:07.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:36:07.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41340aaef0 con 0x7f413413f650 2024-03-29T10:36:07.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903853318 0 0) 0x7f41340aaef0 con 0x7f413413f650 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 >> v1:172.21.15.136:6789/0 conn(0x7f41341457f0 legacy=0x7f413413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 >> v1:172.21.15.117:6789/0 conn(0x7f41340a9d70 legacy=0x7f413414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 monclient: found mon.noname-b 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4134146340 con 0x7f413413f650 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: _finish_auth 0 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4128001670 con 0x7f413413f650 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:07.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: dump: 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4128001890 con 0x7f413413f650 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:07.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 1 -- 172.21.15.117:0/3899181195 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4128001c20 con 0x7f413413f650 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4131ffb640 10 monclient: dump: 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.995+0000 7f4139a4a640 5 monclient: authenticate success, global_id 4310 2024-03-29T10:36:07.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.996+0000 7f4139a4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:07.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.996+0000 7f4139a4a640 4 set_mon_vals no callback set 2024-03-29T10:36:07.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.996+0000 7f4139a4a640 10 monclient: shutdown 2024-03-29T10:36:07.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.996+0000 7f4139a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:07.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.996+0000 7f4139a4a640 1 -- 172.21.15.117:0/3899181195 >> v1:172.21.15.117:6790/0 conn(0x7f413413f650 legacy=0x7f41340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:07.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.997+0000 7f4139a4a640 1 -- 172.21.15.117:0/3899181195 shutdown_connections 2024-03-29T10:36:07.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.997+0000 7f4139a4a640 1 -- 172.21.15.117:0/3899181195 >> 172.21.15.117:0/3899181195 conn(0x7f4134066a00 msgr2=0x7f4134066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:07.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.998+0000 7f4139a4a640 1 -- 172.21.15.117:0/3899181195 shutdown_connections 2024-03-29T10:36:07.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.998+0000 7f4139a4a640 1 -- 172.21.15.117:0/3899181195 wait complete. 2024-03-29T10:36:07.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:07.999+0000 7f4139a4a640 10 monclient: build_initial_monmap 2024-03-29T10:36:07.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.000+0000 7f4139a4a640 10 monclient: monmap: 2024-03-29T10:36:07.985 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.000+0000 7f4139a4a640 1 librados: starting msgr at 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.000+0000 7f4139a4a640 1 librados: starting objecter 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.000+0000 7f4139a4a640 1 Processor -- start 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.001+0000 7f4139a4a640 1 -- start start 2024-03-29T10:36:07.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.001+0000 7f4139a4a640 1 librados: setting wanted keys 2024-03-29T10:36:07.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.001+0000 7f4139a4a640 1 librados: calling monclient init 2024-03-29T10:36:07.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.001+0000 7f4139a4a640 10 monclient: init 2024-03-29T10:36:07.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.002+0000 7f4139a4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:07.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.002+0000 7f4139a4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:36:07.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.002+0000 7f4139a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f41340a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:07.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.002+0000 7f4139a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f413413f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:07.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.005+0000 7f4139a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f41341457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:07.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.007+0000 7f4132ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f41340a9d70 0x7f41341d75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:48437/0 (socket says 172.21.15.117:48437) 2024-03-29T10:36:07.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.008+0000 7f4132ffd640 1 -- 172.21.15.117:0/1885434202 learned_addr learned my addr 172.21.15.117:0/1885434202 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:07.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.008+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413413a9a0 con 0x7f413413f650 2024-03-29T10:36:07.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.008+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413413ac80 con 0x7f41340a9d70 2024-03-29T10:36:07.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.008+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4134139ad0 con 0x7f41341457f0 2024-03-29T10:36:07.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.008+0000 7f4139a4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:07.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.008+0000 7f4139a4a640 10 monclient(hunting): authenticate will time out at 889.270931s 2024-03-29T10:36:07.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.009+0000 7f411bfff640 10 client.?.objecter ms_handle_connect 0x7f41341457f0 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.009+0000 7f411bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.009+0000 7f411bfff640 10 client.?.objecter ms_handle_connect 0x7f413413f650 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.009+0000 7f411bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.009+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328065643 0 0) 0x7f413413a9a0 con 0x7f413413f650 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.009+0000 7f411bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.009+0000 7f411bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4120004eb0 con 0x7f413413f650 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 client.?.objecter ms_handle_connect 0x7f41340a9d70 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287324628 0 0) 0x7f4134139ad0 con 0x7f41341457f0 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:07.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413413a9a0 con 0x7f41341457f0 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920656217 0 0) 0x7f413413ac80 con 0x7f41340a9d70 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.010+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4134139ad0 con 0x7f41340a9d70 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.011+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2586799508 0 0) 0x7f4120004eb0 con 0x7f413413f650 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.011+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f413413ac80 con 0x7f413413f650 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.011+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4128001670 con 0x7f413413f650 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.011+0000 7f411bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.011+0000 7f411bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.011+0000 7f411bfff640 10 monclient(hunting): dump: 2024-03-29T10:36:07.998 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:07.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.011+0000 7f411bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:08.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 586267871 0 0) 0x7f4134139ad0 con 0x7f41340a9d70 2024-03-29T10:36:08.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4120004eb0 con 0x7f41340a9d70 2024-03-29T10:36:08.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4124001780 con 0x7f41340a9d70 2024-03-29T10:36:08.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:08.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 10 monclient(hunting): dump: 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2525930204 0 0) 0x7f413413a9a0 con 0x7f41341457f0 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4134139ad0 con 0x7f41341457f0 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f412c001c50 con 0x7f41341457f0 2024-03-29T10:36:08.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.014+0000 7f411bfff640 10 monclient(hunting): dump: 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.015+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 773642184 0 0) 0x7f4120004eb0 con 0x7f41340a9d70 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.015+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 >> v1:172.21.15.117:6790/0 conn(0x7f41341457f0 legacy=0x7f41341e2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:08.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 >> v1:172.21.15.117:6789/0 conn(0x7f413413f650 legacy=0x7f41341d7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f411bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f411bfff640 1 monclient: found mon.b 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f411bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f411bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41341396f0 con 0x7f41340a9d70 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f411bfff640 10 monclient: _finish_auth 0 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f411bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f411bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f4139a4a640 5 monclient: authenticate success, global_id 4288 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f4139a4a640 10 monclient: _renew_subs 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f4139a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4134138a90 con 0x7f41340a9d70 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f4139a4a640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f4139a4a640 10 monclient: _renew_subs 2024-03-29T10:36:08.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f4139a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f413413b8d0 con 0x7f41340a9d70 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.016+0000 7f4139a4a640 1 librados: init done 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.017+0000 7f4139a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.018+0000 7f4139a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.018+0000 7f4139a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.018+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41240018e0 con 0x7f41340a9d70 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.018+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4134139c60 con 0x7f41340a9d70 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.018+0000 7f411bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.018+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f41240037b0 con 0x7f41340a9d70 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.018+0000 7f411bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.018+0000 7f411bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.018+0000 7f411bfff640 10 monclient: dump: 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:08.007 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:08.008 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:08.008 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:08.008 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:08.008 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:08.008 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:08.008 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:08.008 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:08.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.019+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f41240039d0 con 0x7f41340a9d70 2024-03-29T10:36:08.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.020+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f41240848f0 con 0x7f41340a9d70 2024-03-29T10:36:08.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.020+0000 7f411bfff640 10 client.4288.objecter ms_dispatch 0x7f4134000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:08.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.020+0000 7f411bfff640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:08.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.020+0000 7f411bfff640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:08.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.020+0000 7f411bfff640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T10:36:08.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.019+0000 7f41317fa640 4 set_mon_vals no callback set 2024-03-29T10:36:08.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.021+0000 7f41327fc640 10 monclient: get_auth_request con 0x7f412006e100 auth_method 0 2024-03-29T10:36:08.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.021+0000 7f411bfff640 10 client.4288.objecter ms_handle_connect 0x7f412006e100 2024-03-29T10:36:08.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.026+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41240501d0 con 0x7f41340a9d70 2024-03-29T10:36:08.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.026+0000 7f411bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:08.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.026+0000 7f411bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:08.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.348+0000 7f4139a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:08.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.348+0000 7f4139a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:08.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.348+0000 7f4139a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:08.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.348+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4134139900 con 0x7f41340a9d70 2024-03-29T10:36:08.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.349+0000 7f411bfff640 1 -- 172.21.15.117:0/1885434202 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f4124052100 con 0x7f41340a9d70 2024-03-29T10:36:08.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.349+0000 7f411bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:08.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.349+0000 7f411bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:08.335 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:36:08.335 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:08.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.354+0000 7f4139a4a640 10 librados: watch_flush enter 2024-03-29T10:36:08.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.357+0000 7f4139a4a640 10 librados: watch_flush exit 2024-03-29T10:36:08.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.357+0000 7f4139a4a640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T10:36:08.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.357+0000 7f4139a4a640 10 client.4288.objecter successfully canceled tick 2024-03-29T10:36:08.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.357+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 >> v1:172.21.15.117:6816/376768171 conn(0x7f412006e100 legacy=0x7f41200705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:08.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.357+0000 7f4139a4a640 10 monclient: shutdown 2024-03-29T10:36:08.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.357+0000 7f4139a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:08.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.357+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 >> v1:172.21.15.136:6789/0 conn(0x7f41340a9d70 legacy=0x7f41341d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:08.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.357+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 shutdown_connections 2024-03-29T10:36:08.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.357+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 >> 172.21.15.117:0/1885434202 conn(0x7f4134066a00 msgr2=0x7f41340b4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:08.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.358+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 shutdown_connections 2024-03-29T10:36:08.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.358+0000 7f4139a4a640 1 -- 172.21.15.117:0/1885434202 wait complete. 2024-03-29T10:36:08.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:08.358+0000 7f4139a4a640 1 librados: shutdown 2024-03-29T10:36:08.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:15.170 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:15.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.343+0000 7ff3f50d0640 10 monclient: get_monmap_and_config 2024-03-29T10:36:15.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.343+0000 7ff3f50d0640 10 monclient: build_initial_monmap 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.343+0000 7ff3f50d0640 10 monclient: monmap: 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:36:15.344740+0000 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:36:15.344740+0000 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:36:15.329 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:15.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.347+0000 7ff3f50d0640 1 Processor -- start 2024-03-29T10:36:15.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.347+0000 7ff3f50d0640 1 -- start start 2024-03-29T10:36:15.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.348+0000 7ff3f50d0640 10 monclient: init 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f01457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f013f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f00a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f00aaef0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f0145bd0 con 0x7ff3f013f650 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f0145eb0 con 0x7ff3f01457f0 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:15.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.349+0000 7ff3f50d0640 10 monclient(hunting): authenticate will time out at 896.611417s 2024-03-29T10:36:15.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.354+0000 7ff3ee59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff3f013f650 0x7ff3f00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:45039/0 (socket says 172.21.15.117:45039) 2024-03-29T10:36:15.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.354+0000 7ff3ee59c640 1 -- 172.21.15.117:0/1635109496 learned_addr learned my addr 172.21.15.117:0/1635109496 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:15.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.357+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762622487 0 0) 0x7ff3f00aaef0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.357+0000 7ff3edd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:36:15.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.357+0000 7ff3edd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:15.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.357+0000 7ff3edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:15.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.357+0000 7ff3edd9b640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:36:15.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.357+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8003520 con 0x7ff3f00a9d70 2024-03-29T10:36:15.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.358+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188232338 0 0) 0x7ff3f0145bd0 con 0x7ff3f013f650 2024-03-29T10:36:15.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.358+0000 7ff3edd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:36:15.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.358+0000 7ff3edd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:15.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.358+0000 7ff3edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:15.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.358+0000 7ff3edd9b640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:36:15.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.358+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f00aaef0 con 0x7ff3f013f650 2024-03-29T10:36:15.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4141492793 0 0) 0x7ff3f00aaef0 con 0x7ff3f013f650 2024-03-29T10:36:15.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 >> v1:172.21.15.117:6790/0 conn(0x7ff3f01457f0 legacy=0x7ff3f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:36:15.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 >> v1:172.21.15.117:6789/0 conn(0x7ff3f00a9d70 legacy=0x7ff3f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 1 monclient: found mon.noname-c 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f0146340 con 0x7ff3f013f650 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: _finish_auth 0 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff3e4001670 con 0x7ff3f013f650 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: dump: 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:15.345 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3e4001890 con 0x7ff3f013f650 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 1 -- 172.21.15.117:0/1635109496 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff3e4001c20 con 0x7ff3f013f650 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:15.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.359+0000 7ff3edd9b640 10 monclient: dump: 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:15.347 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:15.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.360+0000 7ff3f50d0640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:36:15.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.360+0000 7ff3f50d0640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:15.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.360+0000 7ff3f50d0640 4 set_mon_vals no callback set 2024-03-29T10:36:15.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.361+0000 7ff3f50d0640 10 monclient: shutdown 2024-03-29T10:36:15.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.361+0000 7ff3f50d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:15.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.361+0000 7ff3f50d0640 1 -- 172.21.15.117:0/1635109496 >> v1:172.21.15.136:6789/0 conn(0x7ff3f013f650 legacy=0x7ff3f00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.361+0000 7ff3f50d0640 1 -- 172.21.15.117:0/1635109496 shutdown_connections 2024-03-29T10:36:15.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.361+0000 7ff3f50d0640 1 -- 172.21.15.117:0/1635109496 >> 172.21.15.117:0/1635109496 conn(0x7ff3f0066a00 msgr2=0x7ff3f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:15.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.362+0000 7ff3f50d0640 1 -- 172.21.15.117:0/1635109496 shutdown_connections 2024-03-29T10:36:15.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.362+0000 7ff3f50d0640 1 -- 172.21.15.117:0/1635109496 wait complete. 2024-03-29T10:36:15.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.363+0000 7ff3f50d0640 10 monclient: build_initial_monmap 2024-03-29T10:36:15.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.363+0000 7ff3f50d0640 10 monclient: monmap: 2024-03-29T10:36:15.350 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:15.350 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:15.350 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:36:15.350 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:36:15.350 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:15.350 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:15.350 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.363+0000 7ff3f50d0640 1 librados: starting msgr at 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.363+0000 7ff3f50d0640 1 librados: starting objecter 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.364+0000 7ff3f50d0640 1 Processor -- start 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.365+0000 7ff3f50d0640 1 -- start start 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.365+0000 7ff3f50d0640 1 librados: setting wanted keys 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.365+0000 7ff3f50d0640 1 librados: calling monclient init 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.365+0000 7ff3f50d0640 10 monclient: init 2024-03-29T10:36:15.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.366+0000 7ff3f50d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:15.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.366+0000 7ff3f50d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:15.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.366+0000 7ff3f50d0640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f00a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:15.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.369+0000 7ff3f50d0640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f01457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.369+0000 7ff3f50d0640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f01387e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.369+0000 7ff3f50d0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f0143180 con 0x7ff3f01387e0 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.369+0000 7ff3f50d0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f01e4b40 con 0x7ff3f01457f0 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.369+0000 7ff3f50d0640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f01e4cd0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.369+0000 7ff3f50d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.369+0000 7ff3f50d0640 10 monclient(hunting): authenticate will time out at 896.631918s 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.370+0000 7ff3eed9d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ff3f00a9d70 0x7ff3f0137f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41791/0 (socket says 172.21.15.117:41791) 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.370+0000 7ff3eed9d640 1 -- 172.21.15.117:0/81663979 learned_addr learned my addr 172.21.15.117:0/81663979 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.370+0000 7ff3d77fe640 10 client.?.objecter ms_handle_connect 0x7ff3f00a9d70 2024-03-29T10:36:15.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.370+0000 7ff3d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:15.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 client.?.objecter ms_handle_connect 0x7ff3f01457f0 2024-03-29T10:36:15.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:15.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708352193 0 0) 0x7ff3f01e4b40 con 0x7ff3f01457f0 2024-03-29T10:36:15.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:15.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:15.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:15.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc004eb0 con 0x7ff3f01457f0 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 client.?.objecter ms_handle_connect 0x7ff3f01387e0 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882633831 0 0) 0x7ff3f01e4cd0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.376+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f01e4b40 con 0x7ff3f00a9d70 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283435630 0 0) 0x7ff3f01e4b40 con 0x7ff3f00a9d70 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f01e4cd0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff3e00016d0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient(hunting): dump: 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2753550760 0 0) 0x7ff3dc004eb0 con 0x7ff3f01457f0 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f01e4b40 con 0x7ff3f01457f0 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff3e40015b0 con 0x7ff3f01457f0 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient(hunting): dump: 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:15.368 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2417542667 0 0) 0x7ff3f01e4cd0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 >> v1:172.21.15.136:6789/0 conn(0x7ff3f01457f0 legacy=0x7ff3f01e4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 >> v1:172.21.15.117:6789/0 conn(0x7ff3f01387e0 legacy=0x7ff3f01d67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 monclient: found mon.c 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f01e4e60 con 0x7ff3f00a9d70 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3f50d0640 5 monclient: authenticate success, global_id 4316 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3f50d0640 10 monclient: _renew_subs 2024-03-29T10:36:15.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3f50d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f013a730 con 0x7ff3f00a9d70 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3f50d0640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3f50d0640 10 monclient: _renew_subs 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3f50d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f01d6ee0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.377+0000 7ff3f50d0640 1 librados: init done 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3f50d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3f50d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3f50d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3e0001be0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f013a460 con 0x7ff3f00a9d70 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff3e0004a00 con 0x7ff3f00a9d70 2024-03-29T10:36:15.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3d77fe640 10 monclient: dump: 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:15.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.379+0000 7ff3ed59a640 4 set_mon_vals no callback set 2024-03-29T10:36:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.381+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7ff3e001a9d0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.382+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7ff3e0084860 con 0x7ff3f00a9d70 2024-03-29T10:36:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.382+0000 7ff3d77fe640 10 client.4316.objecter ms_dispatch 0x7ff3f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.382+0000 7ff3d77fe640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.382+0000 7ff3d77fe640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.382+0000 7ff3d77fe640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T10:36:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.382+0000 7ff3ee59c640 10 monclient: get_auth_request con 0x7ff3dc06de30 auth_method 0 2024-03-29T10:36:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.382+0000 7ff3d77fe640 10 client.4316.objecter ms_handle_connect 0x7ff3dc06de30 2024-03-29T10:36:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.388+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3e00501a0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.388+0000 7ff3d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.388+0000 7ff3d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:15.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.702+0000 7ff3f50d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:15.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.702+0000 7ff3f50d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:15.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.702+0000 7ff3f50d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:15.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.702+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3f013c090 con 0x7ff3f00a9d70 2024-03-29T10:36:15.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.703+0000 7ff3d77fe640 1 -- 172.21.15.117:0/81663979 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7ff3e00501a0 con 0x7ff3f00a9d70 2024-03-29T10:36:15.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.703+0000 7ff3d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:15.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.703+0000 7ff3d77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:15.691 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:36:15.691 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:15.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.709+0000 7ff3f50d0640 10 librados: watch_flush enter 2024-03-29T10:36:15.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.710+0000 7ff3f50d0640 10 librados: watch_flush exit 2024-03-29T10:36:15.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.710+0000 7ff3f50d0640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T10:36:15.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.710+0000 7ff3f50d0640 10 client.4316.objecter successfully canceled tick 2024-03-29T10:36:15.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.710+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 >> v1:172.21.15.117:6816/376768171 conn(0x7ff3dc06de30 legacy=0x7ff3dc0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.710+0000 7ff3f50d0640 10 monclient: shutdown 2024-03-29T10:36:15.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.710+0000 7ff3f50d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:15.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.710+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 >> v1:172.21.15.117:6790/0 conn(0x7ff3f00a9d70 legacy=0x7ff3f0137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.713+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 shutdown_connections 2024-03-29T10:36:15.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.713+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 >> 172.21.15.117:0/81663979 conn(0x7ff3f0066a00 msgr2=0x7ff3f00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:15.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.713+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 shutdown_connections 2024-03-29T10:36:15.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.714+0000 7ff3f50d0640 1 -- 172.21.15.117:0/81663979 wait complete. 2024-03-29T10:36:15.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:15.714+0000 7ff3f50d0640 1 librados: shutdown 2024-03-29T10:36:15.716 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:22.524 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:22.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.690+0000 7f2626251640 10 monclient: get_monmap_and_config 2024-03-29T10:36:22.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.690+0000 7f2626251640 10 monclient: build_initial_monmap 2024-03-29T10:36:22.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.690+0000 7f2626251640 10 monclient: monmap: 2024-03-29T10:36:22.678 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:22.678 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:22.678 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:36:22.691439+0000 2024-03-29T10:36:22.679 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:36:22.691439+0000 2024-03-29T10:36:22.679 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:22.679 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:22.679 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:22.679 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:36:22.679 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:36:22.679 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:22.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.694+0000 7f2626251640 1 Processor -- start 2024-03-29T10:36:22.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.694+0000 7f2626251640 1 -- start start 2024-03-29T10:36:22.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.694+0000 7f2626251640 10 monclient: init 2024-03-29T10:36:22.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.695+0000 7f2626251640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:22.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.695+0000 7f2626251640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:22.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.695+0000 7f2626251640 10 monclient(hunting): picked mon.noname-b con 0x7f26200acf60 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:22.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.695+0000 7f2626251640 10 monclient(hunting): picked mon.noname-a con 0x7f262013ff40 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:22.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.697+0000 7f2626251640 10 monclient(hunting): picked mon.noname-c con 0x7f26201403f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:22.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.697+0000 7f2626251640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26200b4310 con 0x7f262013ff40 2024-03-29T10:36:22.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.697+0000 7f2626251640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2620140800 con 0x7f26201403f0 2024-03-29T10:36:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.697+0000 7f2626251640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26200aec00 con 0x7f26200acf60 2024-03-29T10:36:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.697+0000 7f2626251640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.697+0000 7f2626251640 10 monclient(hunting): authenticate will time out at 903.959526s 2024-03-29T10:36:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.698+0000 7f261ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f26201403f0 0x7f262014bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:47455/0 (socket says 172.21.15.117:47455) 2024-03-29T10:36:22.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.698+0000 7f261ffff640 1 -- 172.21.15.117:0/596641650 learned_addr learned my addr 172.21.15.117:0/596641650 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:22.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.699+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368702596 0 0) 0x7f2620140800 con 0x7f26201403f0 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.699+0000 7f261e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.699+0000 7f261e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.699+0000 7f261e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.699+0000 7f261e7fc640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.700+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2608003520 con 0x7f26201403f0 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012900648 0 0) 0x7f26200b4310 con 0x7f262013ff40 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2620140800 con 0x7f262013ff40 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 561748208 0 0) 0x7f2620140800 con 0x7f262013ff40 2024-03-29T10:36:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 >> v1:172.21.15.117:6790/0 conn(0x7f26200acf60 legacy=0x7f2620139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 >> v1:172.21.15.136:6789/0 conn(0x7f26201403f0 legacy=0x7f262014bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 1 monclient: found mon.noname-a 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26200aeee0 con 0x7f262013ff40 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2610001630 con 0x7f262013ff40 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f261e7fc640 10 monclient: dump: 2024-03-29T10:36:22.693 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:22.693 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:22.693 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:22.693 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:22.693 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:22.693 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:22.693 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:22.693 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:22.693 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:22.694 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f2626251640 5 monclient: authenticate success, global_id 4305 2024-03-29T10:36:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.701+0000 7f2626251640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.702+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2610001880 con 0x7f262013ff40 2024-03-29T10:36:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.702+0000 7f261e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.702+0000 7f261e7fc640 1 -- 172.21.15.117:0/596641650 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2610001c10 con 0x7f262013ff40 2024-03-29T10:36:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.702+0000 7f261e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.702+0000 7f261e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.702+0000 7f261e7fc640 10 monclient: dump: 2024-03-29T10:36:22.695 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:22.695 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:22.695 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:22.695 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.702+0000 7f2626251640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.702+0000 7f2626251640 4 set_mon_vals no callback set 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.703+0000 7f2626251640 10 monclient: shutdown 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.703+0000 7f2626251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.703+0000 7f2626251640 1 -- 172.21.15.117:0/596641650 >> v1:172.21.15.117:6789/0 conn(0x7f262013ff40 legacy=0x7f2620148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:22.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.705+0000 7f2626251640 1 -- 172.21.15.117:0/596641650 shutdown_connections 2024-03-29T10:36:22.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.705+0000 7f2626251640 1 -- 172.21.15.117:0/596641650 >> 172.21.15.117:0/596641650 conn(0x7f26200b35e0 msgr2=0x7f26200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:22.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.707+0000 7f2626251640 1 -- 172.21.15.117:0/596641650 shutdown_connections 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.708+0000 7f2626251640 1 -- 172.21.15.117:0/596641650 wait complete. 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.709+0000 7f2626251640 10 monclient: build_initial_monmap 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.709+0000 7f2626251640 10 monclient: monmap: 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:36:22.705 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.710+0000 7f2626251640 1 librados: starting msgr at 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.710+0000 7f2626251640 1 librados: starting objecter 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.710+0000 7f2626251640 1 Processor -- start 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.710+0000 7f2626251640 1 -- start start 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.710+0000 7f2626251640 1 librados: setting wanted keys 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.710+0000 7f2626251640 1 librados: calling monclient init 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.710+0000 7f2626251640 10 monclient: init 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 10 monclient(hunting): picked mon.noname-b con 0x7f26200acf60 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 10 monclient(hunting): picked mon.noname-a con 0x7f262013ff40 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 10 monclient(hunting): picked mon.noname-c con 0x7f26201403f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262013b5d0 con 0x7f262013ff40 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262013b840 con 0x7f26200acf60 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262013bb20 con 0x7f26201403f0 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.711+0000 7f2626251640 10 monclient(hunting): authenticate will time out at 903.973643s 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.714+0000 7f261ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f26201403f0 0x7f26201315d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:35159/0 (socket says 172.21.15.117:35159) 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.714+0000 7f261ffff640 1 -- 172.21.15.117:0/3065792595 learned_addr learned my addr 172.21.15.117:0/3065792595 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 10 client.?.objecter ms_handle_connect 0x7f26201403f0 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509391611 0 0) 0x7f262013bb20 con 0x7f26201403f0 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f260c004eb0 con 0x7f26201403f0 2024-03-29T10:36:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 10 client.?.objecter ms_handle_connect 0x7f262013ff40 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2876857456 0 0) 0x7f260c004eb0 con 0x7f26201403f0 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.715+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f262013bb20 con 0x7f26201403f0 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2618001b50 con 0x7f26201403f0 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 monclient(hunting): dump: 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:22.708 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169996714 0 0) 0x7f262013b5d0 con 0x7f262013ff40 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f260c004eb0 con 0x7f262013ff40 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 client.?.objecter ms_handle_connect 0x7f26200acf60 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2237812141 0 0) 0x7f262013bb20 con 0x7f26201403f0 2024-03-29T10:36:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 >> v1:172.21.15.136:6789/0 conn(0x7f26200acf60 legacy=0x7f262013d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.716+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 >> v1:172.21.15.117:6789/0 conn(0x7f262013ff40 legacy=0x7f26201c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2607fff640 10 monclient: _finish_hunting 0 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2607fff640 1 monclient: found mon.c 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2607fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2607fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26201ca3c0 con 0x7f26201403f0 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2607fff640 10 monclient: _finish_auth 0 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2607fff640 10 monclient: _check_auth_tickets 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2607fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2626251640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2626251640 10 monclient: _renew_subs 2024-03-29T10:36:22.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2626251640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26201ca610 con 0x7f26201403f0 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2626251640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2626251640 10 monclient: _renew_subs 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2626251640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26201cb190 con 0x7f26201403f0 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.717+0000 7f2626251640 1 librados: init done 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.718+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2618001cb0 con 0x7f26201403f0 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.718+0000 7f2607fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.718+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2618004d40 con 0x7f26201403f0 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.718+0000 7f2607fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.718+0000 7f261dffb640 4 set_mon_vals no callback set 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.718+0000 7f2607fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.718+0000 7f2607fff640 10 monclient: dump: 2024-03-29T10:36:22.711 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.719+0000 7f2626251640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.719+0000 7f2626251640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.719+0000 7f2626251640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:22.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.719+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2600005080 con 0x7f26201403f0 2024-03-29T10:36:22.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.725+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f2618004fc0 con 0x7f26201403f0 2024-03-29T10:36:22.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.726+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f2618084d50 con 0x7f26201403f0 2024-03-29T10:36:22.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.726+0000 7f2607fff640 10 client.4319.objecter ms_dispatch 0x7f2620000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:22.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.726+0000 7f2607fff640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:22.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.726+0000 7f2607fff640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:22.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.726+0000 7f261f7fe640 10 monclient: get_auth_request con 0x7f260c06daa0 auth_method 0 2024-03-29T10:36:22.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.726+0000 7f2607fff640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T10:36:22.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.727+0000 7f2607fff640 10 client.4319.objecter ms_handle_connect 0x7f260c06daa0 2024-03-29T10:36:22.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.731+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2618050630 con 0x7f26201403f0 2024-03-29T10:36:22.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.731+0000 7f2607fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:22.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:22.731+0000 7f2607fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:23.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.053+0000 7f2626251640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:23.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.053+0000 7f2626251640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:23.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.053+0000 7f2626251640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:23.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.053+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2600005330 con 0x7f26201403f0 2024-03-29T10:36:23.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.054+0000 7f2607fff640 1 -- 172.21.15.117:0/3065792595 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f2618052560 con 0x7f26201403f0 2024-03-29T10:36:23.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.054+0000 7f2607fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:23.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.054+0000 7f2607fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:23.040 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:36:23.040 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:23.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.057+0000 7f2626251640 10 librados: watch_flush enter 2024-03-29T10:36:23.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.061+0000 7f2626251640 10 librados: watch_flush exit 2024-03-29T10:36:23.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.061+0000 7f2626251640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T10:36:23.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.061+0000 7f2626251640 10 client.4319.objecter successfully canceled tick 2024-03-29T10:36:23.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.061+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 >> v1:172.21.15.117:6816/376768171 conn(0x7f260c06daa0 legacy=0x7f260c06ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:23.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.061+0000 7f2626251640 10 monclient: shutdown 2024-03-29T10:36:23.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.061+0000 7f2626251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:23.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.061+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 >> v1:172.21.15.117:6790/0 conn(0x7f26201403f0 legacy=0x7f26201315d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:23.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.062+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 shutdown_connections 2024-03-29T10:36:23.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.062+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 >> 172.21.15.117:0/3065792595 conn(0x7f26200b35e0 msgr2=0x7f26200b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:23.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.062+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 shutdown_connections 2024-03-29T10:36:23.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.062+0000 7f2626251640 1 -- 172.21.15.117:0/3065792595 wait complete. 2024-03-29T10:36:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:23.062+0000 7f2626251640 1 librados: shutdown 2024-03-29T10:36:23.065 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:29.872 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:30.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.055+0000 7f601759e640 10 monclient: get_monmap_and_config 2024-03-29T10:36:30.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.055+0000 7f601759e640 10 monclient: build_initial_monmap 2024-03-29T10:36:30.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.055+0000 7f601759e640 10 monclient: monmap: 2024-03-29T10:36:30.040 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:30.040 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:30.041 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:36:30.056586+0000 2024-03-29T10:36:30.041 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:36:30.056586+0000 2024-03-29T10:36:30.041 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:30.041 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:30.041 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:30.041 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:36:30.041 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:36:30.041 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:30.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.059+0000 7f601759e640 1 Processor -- start 2024-03-29T10:36:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.059+0000 7f601759e640 1 -- start start 2024-03-29T10:36:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.059+0000 7f601759e640 10 monclient: init 2024-03-29T10:36:30.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:30.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:30.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 10 monclient(hunting): picked mon.noname-b con 0x7f6010006e30 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 10 monclient(hunting): picked mon.noname-a con 0x7f6010035e00 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 10 monclient(hunting): picked mon.noname-c con 0x7f60100c9670 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6010035970 con 0x7f6010035e00 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6010036280 con 0x7f60100c9670 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6010035640 con 0x7f6010006e30 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.060+0000 7f601759e640 10 monclient(hunting): authenticate will time out at 911.322391s 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.061+0000 7f601659c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f6010006e30 0x7f60100cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:42520/0 (socket says 172.21.15.117:42520) 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.061+0000 7f601659c640 1 -- 172.21.15.117:0/3452169543 learned_addr learned my addr 172.21.15.117:0/3452169543 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.061+0000 7f601559a640 1 -- 172.21.15.117:0/3452169543 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918532617 0 0) 0x7f6010035640 con 0x7f6010006e30 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.061+0000 7f601559a640 10 monclient(hunting): _negotiate 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.061+0000 7f601559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.061+0000 7f601559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.061+0000 7f601559a640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.061+0000 7f601559a640 1 -- 172.21.15.117:0/3452169543 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6000003520 con 0x7f6010006e30 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 1 -- 172.21.15.117:0/3452169543 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3177028456 0 0) 0x7f6000003520 con 0x7f6010006e30 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 1 -- 172.21.15.117:0/3452169543 >> v1:172.21.15.136:6789/0 conn(0x7f60100c9670 legacy=0x7f60100d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 1 -- 172.21.15.117:0/3452169543 >> v1:172.21.15.117:6789/0 conn(0x7f6010035e00 legacy=0x7f60100c8f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 10 monclient: _finish_hunting 0 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 1 monclient: found mon.noname-b 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 1 -- 172.21.15.117:0/3452169543 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6010034e20 con 0x7f6010006e30 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 10 monclient: _finish_auth 0 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 10 monclient: _check_auth_tickets 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 1 -- 172.21.15.117:0/3452169543 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f601804f730 con 0x7f6010006e30 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601559a640 10 monclient: dump: 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601759e640 5 monclient: authenticate success, global_id 4322 2024-03-29T10:36:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.064+0000 7f601759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.066+0000 7f601559a640 1 -- 172.21.15.117:0/3452169543 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6018075f10 con 0x7f6010006e30 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.066+0000 7f601559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.066+0000 7f601559a640 1 -- 172.21.15.117:0/3452169543 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f60180758b0 con 0x7f6010006e30 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.066+0000 7f601559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.066+0000 7f601559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.066+0000 7f601559a640 10 monclient: dump: 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:30.056 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.066+0000 7f601759e640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.066+0000 7f601759e640 4 set_mon_vals no callback set 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.067+0000 7f601759e640 10 monclient: shutdown 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.067+0000 7f601759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.067+0000 7f601759e640 1 -- 172.21.15.117:0/3452169543 >> v1:172.21.15.117:6790/0 conn(0x7f6010006e30 legacy=0x7f60100cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.067+0000 7f601759e640 1 -- 172.21.15.117:0/3452169543 shutdown_connections 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.067+0000 7f601759e640 1 -- 172.21.15.117:0/3452169543 >> 172.21.15.117:0/3452169543 conn(0x7f6010005c00 msgr2=0x7f6010006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.067+0000 7f601759e640 1 -- 172.21.15.117:0/3452169543 shutdown_connections 2024-03-29T10:36:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.067+0000 7f601759e640 1 -- 172.21.15.117:0/3452169543 wait complete. 2024-03-29T10:36:30.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.069+0000 7f601759e640 10 monclient: build_initial_monmap 2024-03-29T10:36:30.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.069+0000 7f601759e640 10 monclient: monmap: 2024-03-29T10:36:30.059 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.069+0000 7f601759e640 1 librados: starting msgr at 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.069+0000 7f601759e640 1 librados: starting objecter 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.070+0000 7f601759e640 1 Processor -- start 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.071+0000 7f601759e640 1 -- start start 2024-03-29T10:36:30.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.071+0000 7f601759e640 1 librados: setting wanted keys 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.071+0000 7f601759e640 1 librados: calling monclient init 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.071+0000 7f601759e640 10 monclient: init 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.073+0000 7f601759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.073+0000 7f601759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f601759e640 10 monclient(hunting): picked mon.noname-a con 0x7f6010006e30 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f601759e640 10 monclient(hunting): picked mon.noname-c con 0x7f6010035e00 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f601759e640 10 monclient(hunting): picked mon.noname-b con 0x7f60100c9670 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f601759e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60101640e0 con 0x7f6010006e30 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f601759e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60100cdcd0 con 0x7f60100c9670 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f601759e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6010031ed0 con 0x7f6010035e00 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f601759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f601759e640 10 monclient(hunting): authenticate will time out at 911.336417s 2024-03-29T10:36:30.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f6015d9b640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f6010035e00 0x7f601002f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44024/0 (socket says 172.21.15.117:44024) 2024-03-29T10:36:30.062 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.074+0000 7f6015d9b640 1 -- 172.21.15.117:0/3266662700 learned_addr learned my addr 172.21.15.117:0/3266662700 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.076+0000 7f6006ffd640 10 client.?.objecter ms_handle_connect 0x7f6010035e00 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.076+0000 7f6006ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.076+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827211013 0 0) 0x7f6010031ed0 con 0x7f6010035e00 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.076+0000 7f6006ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.076+0000 7f6006ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.076+0000 7f6006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.076+0000 7f6006ffd640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.076+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff8004eb0 con 0x7f6010035e00 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.078+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 563653154 0 0) 0x7f5ff8004eb0 con 0x7f6010035e00 2024-03-29T10:36:30.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.078+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6010031ed0 con 0x7f6010035e00 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.078+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6008001960 con 0x7f6010035e00 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.078+0000 7f6006ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.078+0000 7f6006ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.078+0000 7f6006ffd640 10 monclient(hunting): dump: 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:30.065 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:30.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.078+0000 7f6006ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:30.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.079+0000 7f6006ffd640 10 client.?.objecter ms_handle_connect 0x7f6010006e30 2024-03-29T10:36:30.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.079+0000 7f6006ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:30.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.079+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2351504473 0 0) 0x7f6010031ed0 con 0x7f6010035e00 2024-03-29T10:36:30.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.079+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 >> v1:172.21.15.136:6789/0 conn(0x7f60100c9670 legacy=0x7f60100ce240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:36:30.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.083+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 >> v1:172.21.15.117:6789/0 conn(0x7f6010006e30 legacy=0x7f6010164600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:30.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f6006ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:36:30.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f6006ffd640 1 monclient: found mon.c 2024-03-29T10:36:30.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f6006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:30.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f6006ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:30.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6010032060 con 0x7f6010035e00 2024-03-29T10:36:30.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f6006ffd640 10 monclient: _finish_auth 0 2024-03-29T10:36:30.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f6006ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:36:30.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f6006ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f601759e640 5 monclient: authenticate success, global_id 4325 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f601759e640 10 monclient: _renew_subs 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f601759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6010164d10 con 0x7f6010035e00 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f601759e640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f601759e640 10 monclient: _renew_subs 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f601759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f601002ffb0 con 0x7f6010035e00 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.084+0000 7f601759e640 1 librados: init done 2024-03-29T10:36:30.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.086+0000 7f601759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:30.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.086+0000 7f601759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:30.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.086+0000 7f601759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:30.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.086+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6010007b90 con 0x7f6010035e00 2024-03-29T10:36:30.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.088+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6008001ac0 con 0x7f6010035e00 2024-03-29T10:36:30.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.088+0000 7f6006ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.088+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6008001d80 con 0x7f6010035e00 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.088+0000 7f6006ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.088+0000 7f6014d99640 4 set_mon_vals no callback set 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.089+0000 7f6006ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.089+0000 7f6006ffd640 10 monclient: dump: 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:30.074 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:30.075 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:30.075 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:30.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.089+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f6008003960 con 0x7f6010035e00 2024-03-29T10:36:30.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.090+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f6008084990 con 0x7f6010035e00 2024-03-29T10:36:30.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.090+0000 7f6006ffd640 10 client.4325.objecter ms_dispatch 0x7f6018000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:30.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.090+0000 7f6006ffd640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:30.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.090+0000 7f6006ffd640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:30.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.090+0000 7f6006ffd640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T10:36:30.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.090+0000 7f601659c640 10 monclient: get_auth_request con 0x7f5ff806db70 auth_method 0 2024-03-29T10:36:30.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.091+0000 7f6006ffd640 10 client.4325.objecter ms_handle_connect 0x7f5ff806db70 2024-03-29T10:36:30.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.099+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6008050270 con 0x7f6010035e00 2024-03-29T10:36:30.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.099+0000 7f6006ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:30.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.099+0000 7f6006ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:30.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.465+0000 7f601759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:30.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.465+0000 7f601759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:30.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.465+0000 7f601759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:30.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.465+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6010005120 con 0x7f6010035e00 2024-03-29T10:36:30.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.466+0000 7f6006ffd640 1 -- 172.21.15.117:0/3266662700 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f60080521a0 con 0x7f6010035e00 2024-03-29T10:36:30.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.466+0000 7f6006ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:30.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.466+0000 7f6006ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:30.452 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:36:30.452 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:30.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 10 librados: watch_flush enter 2024-03-29T10:36:30.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 10 librados: watch_flush exit 2024-03-29T10:36:30.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T10:36:30.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 10 client.4325.objecter successfully canceled tick 2024-03-29T10:36:30.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 >> v1:172.21.15.117:6816/376768171 conn(0x7f5ff806db70 legacy=0x7f5ff8070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:30.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 10 monclient: shutdown 2024-03-29T10:36:30.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:30.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 >> v1:172.21.15.117:6790/0 conn(0x7f6010035e00 legacy=0x7f601002f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:30.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 shutdown_connections 2024-03-29T10:36:30.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 >> 172.21.15.117:0/3266662700 conn(0x7f6010005c00 msgr2=0x7f60100c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:30.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.470+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 shutdown_connections 2024-03-29T10:36:30.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.471+0000 7f601759e640 1 -- 172.21.15.117:0/3266662700 wait complete. 2024-03-29T10:36:30.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:30.471+0000 7f601759e640 1 librados: shutdown 2024-03-29T10:36:30.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:37.278 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:37.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.447+0000 7f0cc7c8d640 10 monclient: get_monmap_and_config 2024-03-29T10:36:37.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.447+0000 7f0cc7c8d640 10 monclient: build_initial_monmap 2024-03-29T10:36:37.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.447+0000 7f0cc7c8d640 10 monclient: monmap: 2024-03-29T10:36:37.433 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:37.433 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:37.433 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:36:37.449463+0000 2024-03-29T10:36:37.433 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:36:37.449463+0000 2024-03-29T10:36:37.433 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:37.433 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:37.434 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:37.434 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:36:37.434 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:36:37.434 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:37.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.448+0000 7f0cc7c8d640 1 Processor -- start 2024-03-29T10:36:37.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.448+0000 7f0cc7c8d640 1 -- start start 2024-03-29T10:36:37.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.448+0000 7f0cc7c8d640 10 monclient: init 2024-03-29T10:36:37.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:37.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:37.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f0cc01457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:37.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc013f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:37.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f0cc00a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:37.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc00aaef0 con 0x7f0cc013f650 2024-03-29T10:36:37.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc0145bd0 con 0x7f0cc00a9d70 2024-03-29T10:36:37.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc0145eb0 con 0x7f0cc01457f0 2024-03-29T10:36:37.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:37.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.449+0000 7f0cc7c8d640 10 monclient(hunting): authenticate will time out at 918.713030s 2024-03-29T10:36:37.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc5228640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0cc013f650 0x7f0cc00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:40786/0 (socket says 172.21.15.117:40786) 2024-03-29T10:36:37.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc5228640 1 -- 172.21.15.117:0/2874116401 learned_addr learned my addr 172.21.15.117:0/2874116401 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:37.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103794551 0 0) 0x7f0cc00aaef0 con 0x7f0cc013f650 2024-03-29T10:36:37.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 10 monclient(hunting): _negotiate 2024-03-29T10:36:37.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:37.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:37.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cac003520 con 0x7f0cc013f650 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410866837 0 0) 0x7f0cc0145bd0 con 0x7f0cc00a9d70 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 10 monclient(hunting): _negotiate 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.450+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc00aaef0 con 0x7f0cc00a9d70 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154672189 0 0) 0x7f0cac003520 con 0x7f0cc013f650 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 >> v1:172.21.15.117:6790/0 conn(0x7f0cc01457f0 legacy=0x7f0cc013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 >> v1:172.21.15.136:6789/0 conn(0x7f0cc00a9d70 legacy=0x7f0cc014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: _finish_hunting 0 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 1 monclient: found mon.noname-a 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc0146340 con 0x7f0cc013f650 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: _finish_auth 0 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: _check_auth_tickets 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0cbc001690 con 0x7f0cc013f650 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: dump: 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:37.439 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc7c8d640 5 monclient: authenticate success, global_id 4314 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc7c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cbc0017f0 con 0x7f0cc013f650 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 1 -- 172.21.15.117:0/2874116401 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0cbc001c00 con 0x7f0cc013f650 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc4a27640 10 monclient: dump: 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc7c8d640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.451+0000 7f0cc7c8d640 4 set_mon_vals no callback set 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.452+0000 7f0cc7c8d640 10 monclient: shutdown 2024-03-29T10:36:37.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.452+0000 7f0cc7c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:37.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.452+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2874116401 >> v1:172.21.15.117:6789/0 conn(0x7f0cc013f650 legacy=0x7f0cc00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.452+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2874116401 shutdown_connections 2024-03-29T10:36:37.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.452+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2874116401 >> 172.21.15.117:0/2874116401 conn(0x7f0cc0066a00 msgr2=0x7f0cc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:37.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.452+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2874116401 shutdown_connections 2024-03-29T10:36:37.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.452+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2874116401 wait complete. 2024-03-29T10:36:37.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.453+0000 7f0cc7c8d640 10 monclient: build_initial_monmap 2024-03-29T10:36:37.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.453+0000 7f0cc7c8d640 10 monclient: monmap: 2024-03-29T10:36:37.448 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:37.448 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:37.448 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:36:37.448 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.454+0000 7f0cc7c8d640 1 librados: starting msgr at 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.454+0000 7f0cc7c8d640 1 librados: starting objecter 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.454+0000 7f0cc7c8d640 1 Processor -- start 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.454+0000 7f0cc7c8d640 1 -- start start 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.454+0000 7f0cc7c8d640 1 librados: setting wanted keys 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.454+0000 7f0cc7c8d640 1 librados: calling monclient init 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.454+0000 7f0cc7c8d640 10 monclient: init 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:37.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f0cc00a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f0cc013f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc5a29640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f0cc00a9d70 0x7f0cc01cb5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:37100/0 (socket says 172.21.15.117:37100) 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc01457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc01cc0c0 con 0x7f0cc01457f0 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc01cf100 con 0x7f0cc013f650 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc01cbcc0 con 0x7f0cc00a9d70 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc622a640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0cc01457f0 0x7f0cc01d78d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:41644/0 (socket says 172.21.15.117:41644) 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc7c8d640 10 monclient(hunting): authenticate will time out at 918.718828s 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cc622a640 1 -- 172.21.15.117:0/2011802098 learned_addr learned my addr 172.21.15.117:0/2011802098 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cb27fc640 10 client.?.objecter ms_handle_connect 0x7f0cc013f650 2024-03-29T10:36:37.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.455+0000 7f0cb27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 client.?.objecter ms_handle_connect 0x7f0cc00a9d70 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 client.?.objecter ms_handle_connect 0x7f0cc01457f0 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35399790 0 0) 0x7f0cc01cf100 con 0x7f0cc013f650 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca4004eb0 con 0x7f0cc013f650 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558879201 0 0) 0x7f0cc01cbcc0 con 0x7f0cc00a9d70 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:37.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.456+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc01cf100 con 0x7f0cc00a9d70 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963098420 0 0) 0x7f0cc01cc0c0 con 0x7f0cc01457f0 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc01cbcc0 con 0x7f0cc01457f0 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4080295920 0 0) 0x7f0cc01cf100 con 0x7f0cc00a9d70 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cc01cc0c0 con 0x7f0cc00a9d70 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0cb4001750 con 0x7f0cc00a9d70 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): dump: 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:37.452 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1926320499 0 0) 0x7f0ca4004eb0 con 0x7f0cc013f650 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cc01cf100 con 0x7f0cc013f650 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0cbc001b00 con 0x7f0cc013f650 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient(hunting): dump: 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1980811419 0 0) 0x7f0cc01cc0c0 con 0x7f0cc00a9d70 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 >> v1:172.21.15.136:6789/0 conn(0x7f0cc013f650 legacy=0x7f0cc0137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 >> v1:172.21.15.117:6789/0 conn(0x7f0cc01457f0 legacy=0x7f0cc01d78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 monclient: found mon.c 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc013b870 con 0x7f0cc00a9d70 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient: _finish_auth 0 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.457+0000 7f0cb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cc7c8d640 5 monclient: authenticate success, global_id 4331 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cc7c8d640 10 monclient: _renew_subs 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cc7c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc013bb40 con 0x7f0cc00a9d70 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cc7c8d640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cc7c8d640 10 monclient: _renew_subs 2024-03-29T10:36:37.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cc7c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc0138630 con 0x7f0cc00a9d70 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cc7c8d640 1 librados: init done 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cb4001c40 con 0x7f0cc00a9d70 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0cb40037b0 con 0x7f0cc00a9d70 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.458+0000 7f0cb27fc640 10 monclient: dump: 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:37.456 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.459+0000 7f0cb3fff640 4 set_mon_vals no callback set 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.459+0000 7f0cc7c8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.459+0000 7f0cc7c8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.459+0000 7f0cc7c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.460+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca8005080 con 0x7f0cc00a9d70 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.461+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f0cb4003910 con 0x7f0cc00a9d70 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.461+0000 7f0cc5228640 10 monclient: get_auth_request con 0x7f0ca406def0 auth_method 0 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.462+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f0cb4084840 con 0x7f0cc00a9d70 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.462+0000 7f0cb27fc640 10 client.4331.objecter ms_dispatch 0x7f0cc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.462+0000 7f0cb27fc640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:37.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.462+0000 7f0cb27fc640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:37.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.462+0000 7f0cb27fc640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T10:36:37.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.462+0000 7f0cb27fc640 10 client.4331.objecter ms_handle_connect 0x7f0ca406def0 2024-03-29T10:36:37.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.470+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0cb4050180 con 0x7f0cc00a9d70 2024-03-29T10:36:37.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.470+0000 7f0cb27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:37.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.470+0000 7f0cb27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:37.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.842+0000 7f0cc7c8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:37.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.842+0000 7f0cc7c8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:37.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.842+0000 7f0cc7c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:37.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.842+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ca8005330 con 0x7f0cc00a9d70 2024-03-29T10:36:37.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.843+0000 7f0cb27fc640 1 -- 172.21.15.117:0/2011802098 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f0cb4083ef0 con 0x7f0cc00a9d70 2024-03-29T10:36:37.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.843+0000 7f0cb27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:37.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.843+0000 7f0cb27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:37.829 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:36:37.829 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:37.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.847+0000 7f0cc7c8d640 10 librados: watch_flush enter 2024-03-29T10:36:37.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.847+0000 7f0cc7c8d640 10 librados: watch_flush exit 2024-03-29T10:36:37.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.847+0000 7f0cc7c8d640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T10:36:37.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.847+0000 7f0cc7c8d640 10 client.4331.objecter successfully canceled tick 2024-03-29T10:36:37.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.847+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 >> v1:172.21.15.117:6816/376768171 conn(0x7f0ca406def0 legacy=0x7f0ca40703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.847+0000 7f0cc7c8d640 10 monclient: shutdown 2024-03-29T10:36:37.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.847+0000 7f0cc7c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:37.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.847+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 >> v1:172.21.15.117:6790/0 conn(0x7f0cc00a9d70 legacy=0x7f0cc01cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.848+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 shutdown_connections 2024-03-29T10:36:37.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.848+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 >> 172.21.15.117:0/2011802098 conn(0x7f0cc0066a00 msgr2=0x7f0cc0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:37.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.848+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 shutdown_connections 2024-03-29T10:36:37.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.848+0000 7f0cc7c8d640 1 -- 172.21.15.117:0/2011802098 wait complete. 2024-03-29T10:36:37.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:37.848+0000 7f0cc7c8d640 1 librados: shutdown 2024-03-29T10:36:37.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:44.656 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:44.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.818+0000 7feace033640 10 monclient: get_monmap_and_config 2024-03-29T10:36:44.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.818+0000 7feace033640 10 monclient: build_initial_monmap 2024-03-29T10:36:44.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.819+0000 7feace033640 10 monclient: monmap: 2024-03-29T10:36:44.809 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:44.809 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:44.809 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:36:44.819891+0000 2024-03-29T10:36:44.809 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:36:44.819891+0000 2024-03-29T10:36:44.809 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.820+0000 7feace033640 1 Processor -- start 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.820+0000 7feace033640 1 -- start start 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.820+0000 7feace033640 10 monclient: init 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 10 monclient(hunting): picked mon.noname-b con 0x7feac806d9d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 10 monclient(hunting): picked mon.noname-c con 0x7feac813ef30 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 10 monclient(hunting): picked mon.noname-a con 0x7feac80b07d0 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac80aed10 con 0x7feac80b07d0 2024-03-29T10:36:44.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac80b0c40 con 0x7feac813ef30 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac80ab1b0 con 0x7feac806d9d0 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feace033640 10 monclient(hunting): authenticate will time out at 926.083572s 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feac77fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7feac806d9d0 0x7feac813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:51074/0 (socket says 172.21.15.117:51074) 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.821+0000 7feac77fe640 1 -- 172.21.15.117:0/3576800504 learned_addr learned my addr 172.21.15.117:0/3576800504 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.822+0000 7feac67fc640 1 -- 172.21.15.117:0/3576800504 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391650893 0 0) 0x7feac80ab1b0 con 0x7feac806d9d0 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.822+0000 7feac67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.822+0000 7feac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.822+0000 7feac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.822+0000 7feac67fc640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:36:44.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.822+0000 7feac67fc640 1 -- 172.21.15.117:0/3576800504 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab0005730 con 0x7feac806d9d0 2024-03-29T10:36:44.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.825+0000 7feac67fc640 1 -- 172.21.15.117:0/3576800504 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2580486375 0 0) 0x7feab0005730 con 0x7feac806d9d0 2024-03-29T10:36:44.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.825+0000 7feac67fc640 1 -- 172.21.15.117:0/3576800504 >> v1:172.21.15.136:6789/0 conn(0x7feac813ef30 legacy=0x7feac80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:36:44.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.825+0000 7feac67fc640 1 -- 172.21.15.117:0/3576800504 >> v1:172.21.15.117:6789/0 conn(0x7feac80b07d0 legacy=0x7feac814ab80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:36:44.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 1 monclient: found mon.noname-b 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 1 -- 172.21.15.117:0/3576800504 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac80ab490 con 0x7feac806d9d0 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 10 monclient: _finish_auth 0 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feace033640 5 monclient: authenticate success, global_id 4334 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feace033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 1 -- 172.21.15.117:0/3576800504 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7feab8001690 con 0x7feac806d9d0 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 10 monclient: dump: 2024-03-29T10:36:44.813 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feace033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 1 -- 172.21.15.117:0/3576800504 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feab8001920 con 0x7feac806d9d0 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feac67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feace033640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.827+0000 7feace033640 4 set_mon_vals no callback set 2024-03-29T10:36:44.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.828+0000 7feace033640 10 monclient: shutdown 2024-03-29T10:36:44.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.828+0000 7feace033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:44.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.828+0000 7feace033640 1 -- 172.21.15.117:0/3576800504 >> v1:172.21.15.117:6790/0 conn(0x7feac806d9d0 legacy=0x7feac813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:44.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.829+0000 7feace033640 1 -- 172.21.15.117:0/3576800504 shutdown_connections 2024-03-29T10:36:44.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.829+0000 7feace033640 1 -- 172.21.15.117:0/3576800504 >> 172.21.15.117:0/3576800504 conn(0x7feac806b660 msgr2=0x7feac806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:44.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.830+0000 7feace033640 1 -- 172.21.15.117:0/3576800504 shutdown_connections 2024-03-29T10:36:44.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.830+0000 7feace033640 1 -- 172.21.15.117:0/3576800504 wait complete. 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.831+0000 7feace033640 10 monclient: build_initial_monmap 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.831+0000 7feace033640 10 monclient: monmap: 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:36:44.817 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.831+0000 7feace033640 1 librados: starting msgr at 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.831+0000 7feace033640 1 librados: starting objecter 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.831+0000 7feace033640 1 Processor -- start 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.832+0000 7feace033640 1 -- start start 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.832+0000 7feace033640 1 librados: setting wanted keys 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.832+0000 7feace033640 1 librados: calling monclient init 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.832+0000 7feace033640 10 monclient: init 2024-03-29T10:36:44.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:44.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:44.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 10 monclient(hunting): picked mon.noname-b con 0x7feac806d9d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:44.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 10 monclient(hunting): picked mon.noname-c con 0x7feac80b07d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:44.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 10 monclient(hunting): picked mon.noname-a con 0x7feac813ef30 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:44.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac806f640 con 0x7feac813ef30 2024-03-29T10:36:44.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac806ea10 con 0x7feac806d9d0 2024-03-29T10:36:44.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac806f8b0 con 0x7feac80b07d0 2024-03-29T10:36:44.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:44.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.833+0000 7feace033640 10 monclient(hunting): authenticate will time out at 926.095408s 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.836+0000 7feac77fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7feac806d9d0 0x7feac806e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:53725/0 (socket says 172.21.15.117:53725) 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.836+0000 7feac77fe640 1 -- 172.21.15.117:0/2496647487 learned_addr learned my addr 172.21.15.117:0/2496647487 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 client.?.objecter ms_handle_connect 0x7feac806d9d0 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696916361 0 0) 0x7feac806ea10 con 0x7feac806d9d0 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:36:44.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab4004eb0 con 0x7feac806d9d0 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 client.?.objecter ms_handle_connect 0x7feac80b07d0 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3423299146 0 0) 0x7feab4004eb0 con 0x7feac806d9d0 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feac806ea10 con 0x7feac806d9d0 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7feab8004a00 con 0x7feac806d9d0 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 monclient(hunting): dump: 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:44.824 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:44.825 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:44.825 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:44.825 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:44.825 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:44.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.837+0000 7feaaffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.839+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2517029294 0 0) 0x7feac806ea10 con 0x7feac806d9d0 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.839+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 >> v1:172.21.15.117:6790/0 conn(0x7feac80b07d0 legacy=0x7feac8070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.839+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 >> v1:172.21.15.117:6789/0 conn(0x7feac813ef30 legacy=0x7feac81dd520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feaaffff640 10 monclient: _finish_hunting 0 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feaaffff640 1 monclient: found mon.b 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feaaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feaaffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac8138820 con 0x7feac806d9d0 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feaaffff640 10 monclient: _finish_auth 0 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feaaffff640 10 monclient: _check_auth_tickets 2024-03-29T10:36:44.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feaaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:44.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feace033640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:36:44.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feace033640 10 monclient: _renew_subs 2024-03-29T10:36:44.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feace033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:44.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.840+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac8138190 con 0x7feac806d9d0 2024-03-29T10:36:44.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.841+0000 7feace033640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:44.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.841+0000 7feace033640 10 monclient: _renew_subs 2024-03-29T10:36:44.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.841+0000 7feace033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:44.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.841+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feac8070980 con 0x7feac806d9d0 2024-03-29T10:36:44.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.841+0000 7feace033640 1 librados: init done 2024-03-29T10:36:44.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.843+0000 7feace033640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:44.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.843+0000 7feace033640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:44.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.843+0000 7feace033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:44.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.843+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac8138530 con 0x7feac806d9d0 2024-03-29T10:36:44.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.844+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feab8001b10 con 0x7feac806d9d0 2024-03-29T10:36:44.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.844+0000 7feaaffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:44.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.844+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7feab8004fb0 con 0x7feac806d9d0 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.844+0000 7feaaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.844+0000 7feaaffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.844+0000 7feaaffff640 10 monclient: dump: 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:44.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.844+0000 7feac57fa640 4 set_mon_vals no callback set 2024-03-29T10:36:44.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.847+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7feab801af80 con 0x7feac806d9d0 2024-03-29T10:36:44.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.848+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7feab8084d00 con 0x7feac806d9d0 2024-03-29T10:36:44.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.848+0000 7feaaffff640 10 client.4306.objecter ms_dispatch 0x7feac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:44.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.848+0000 7feaaffff640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:44.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.848+0000 7feaaffff640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:44.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.848+0000 7feaaffff640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T10:36:44.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.848+0000 7feac6ffd640 10 monclient: get_auth_request con 0x7feab406dc80 auth_method 0 2024-03-29T10:36:44.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.849+0000 7feaaffff640 10 client.4306.objecter ms_handle_connect 0x7feab406dc80 2024-03-29T10:36:44.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.855+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feab80505e0 con 0x7feac806d9d0 2024-03-29T10:36:44.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.855+0000 7feaaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:44.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:44.855+0000 7feaaffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:45.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.187+0000 7feace033640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:45.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.187+0000 7feace033640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:45.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.187+0000 7feace033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:45.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.187+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feac8138320 con 0x7feac806d9d0 2024-03-29T10:36:45.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.188+0000 7feaaffff640 1 -- 172.21.15.117:0/2496647487 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7feab8052510 con 0x7feac806d9d0 2024-03-29T10:36:45.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.188+0000 7feaaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:45.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.188+0000 7feaaffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:45.175 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:36:45.175 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:45.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.192+0000 7feace033640 10 librados: watch_flush enter 2024-03-29T10:36:45.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.192+0000 7feace033640 10 librados: watch_flush exit 2024-03-29T10:36:45.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.192+0000 7feace033640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T10:36:45.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.192+0000 7feace033640 10 client.4306.objecter successfully canceled tick 2024-03-29T10:36:45.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.193+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 >> v1:172.21.15.117:6816/376768171 conn(0x7feab406dc80 legacy=0x7feab4070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:45.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.193+0000 7feace033640 10 monclient: shutdown 2024-03-29T10:36:45.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.193+0000 7feace033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:45.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.193+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 >> v1:172.21.15.136:6789/0 conn(0x7feac806d9d0 legacy=0x7feac806e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:45.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.193+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 shutdown_connections 2024-03-29T10:36:45.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.193+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 >> 172.21.15.117:0/2496647487 conn(0x7feac806b660 msgr2=0x7feac80adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:45.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.193+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 shutdown_connections 2024-03-29T10:36:45.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.193+0000 7feace033640 1 -- 172.21.15.117:0/2496647487 wait complete. 2024-03-29T10:36:45.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:45.193+0000 7feace033640 1 librados: shutdown 2024-03-29T10:36:45.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:52.001 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:52.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.145+0000 7fe8a86a8640 10 monclient: get_monmap_and_config 2024-03-29T10:36:52.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.145+0000 7fe8a86a8640 10 monclient: build_initial_monmap 2024-03-29T10:36:52.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.145+0000 7fe8a86a8640 10 monclient: monmap: 2024-03-29T10:36:52.130 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:52.130 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:52.130 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:36:52.146502+0000 2024-03-29T10:36:52.130 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:36:52.146502+0000 2024-03-29T10:36:52.130 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:52.130 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:52.131 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:52.131 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:36:52.131 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:36:52.131 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:52.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.146+0000 7fe8a86a8640 1 Processor -- start 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.146+0000 7fe8a86a8640 1 -- start start 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.146+0000 7fe8a86a8640 10 monclient: init 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe8a01457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe8a013f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe8a00a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a00aaef0 con 0x7fe8a01457f0 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a0145bd0 con 0x7fe8a013f650 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a0145eb0 con 0x7fe8a00a9d70 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a86a8640 10 monclient(hunting): authenticate will time out at 933.409585s 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a5c43640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe8a013f650 0x7fe8a00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:41326/0 (socket says 172.21.15.117:41326) 2024-03-29T10:36:52.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a5c43640 1 -- 172.21.15.117:0/3646794073 learned_addr learned my addr 172.21.15.117:0/3646794073 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:52.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a5442640 1 -- 172.21.15.117:0/3646794073 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772437829 0 0) 0x7fe8a0145eb0 con 0x7fe8a00a9d70 2024-03-29T10:36:52.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a5442640 10 monclient(hunting): _negotiate 2024-03-29T10:36:52.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a5442640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:52.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a5442640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:52.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a5442640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:36:52.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.147+0000 7fe8a5442640 1 -- 172.21.15.117:0/3646794073 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe88c003520 con 0x7fe8a00a9d70 2024-03-29T10:36:52.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.148+0000 7fe8a5442640 1 -- 172.21.15.117:0/3646794073 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2209208444 0 0) 0x7fe88c003520 con 0x7fe8a00a9d70 2024-03-29T10:36:52.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.148+0000 7fe8a5442640 1 -- 172.21.15.117:0/3646794073 >> v1:172.21.15.136:6789/0 conn(0x7fe8a013f650 legacy=0x7fe8a00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.148+0000 7fe8a5442640 1 -- 172.21.15.117:0/3646794073 >> v1:172.21.15.117:6789/0 conn(0x7fe8a01457f0 legacy=0x7fe8a013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: _finish_hunting 0 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 1 monclient: found mon.noname-b 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 1 -- 172.21.15.117:0/3646794073 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a0146340 con 0x7fe8a00a9d70 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: _finish_auth 0 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: _check_auth_tickets 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 1 -- 172.21.15.117:0/3646794073 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe898001a10 con 0x7fe8a00a9d70 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: dump: 2024-03-29T10:36:52.136 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a86a8640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a86a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 1 -- 172.21.15.117:0/3646794073 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe898001be0 con 0x7fe8a00a9d70 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a86a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:52.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 1 -- 172.21.15.117:0/3646794073 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe898002030 con 0x7fe8a00a9d70 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a86a8640 4 set_mon_vals no callback set 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.151+0000 7fe8a5442640 10 monclient: dump: 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:52.138 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.152+0000 7fe8a86a8640 10 monclient: shutdown 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.152+0000 7fe8a86a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.152+0000 7fe8a86a8640 1 -- 172.21.15.117:0/3646794073 >> v1:172.21.15.117:6790/0 conn(0x7fe8a00a9d70 legacy=0x7fe8a014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.152+0000 7fe8a86a8640 1 -- 172.21.15.117:0/3646794073 shutdown_connections 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.152+0000 7fe8a86a8640 1 -- 172.21.15.117:0/3646794073 >> 172.21.15.117:0/3646794073 conn(0x7fe8a0066a00 msgr2=0x7fe8a0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.153+0000 7fe8a86a8640 1 -- 172.21.15.117:0/3646794073 shutdown_connections 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.153+0000 7fe8a86a8640 1 -- 172.21.15.117:0/3646794073 wait complete. 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.154+0000 7fe8a86a8640 10 monclient: build_initial_monmap 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.154+0000 7fe8a86a8640 10 monclient: monmap: 2024-03-29T10:36:52.140 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.154+0000 7fe8a86a8640 1 librados: starting msgr at 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.154+0000 7fe8a86a8640 1 librados: starting objecter 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.154+0000 7fe8a86a8640 1 Processor -- start 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.155+0000 7fe8a86a8640 1 -- start start 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.155+0000 7fe8a86a8640 1 librados: setting wanted keys 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.155+0000 7fe8a86a8640 1 librados: calling monclient init 2024-03-29T10:36:52.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.155+0000 7fe8a86a8640 10 monclient: init 2024-03-29T10:36:52.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.155+0000 7fe8a86a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:52.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.155+0000 7fe8a86a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:52.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.155+0000 7fe8a86a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe8a00a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.155+0000 7fe8a86a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe8a013f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe8a86a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe8a01457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe8a86a8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a0071990 con 0x7fe8a01457f0 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe8a86a8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a0071c70 con 0x7fe8a013f650 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe8a86a8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a0070480 con 0x7fe8a00a9d70 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe8a86a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe8a86a8640 10 monclient(hunting): authenticate will time out at 933.418146s 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe8a6444640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe8a00a9d70 0x7fe8a01deb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:49388/0 (socket says 172.21.15.117:49388) 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe8a6444640 1 -- 172.21.15.117:0/2521194530 learned_addr learned my addr 172.21.15.117:0/2521194530 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 client.?.objecter ms_handle_connect 0x7fe8a00a9d70 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 client.?.objecter ms_handle_connect 0x7fe8a013f650 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:52.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768447895 0 0) 0x7fe8a0070480 con 0x7fe8a00a9d70 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe884004eb0 con 0x7fe8a00a9d70 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243569486 0 0) 0x7fe8a0071c70 con 0x7fe8a013f650 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.156+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a0070480 con 0x7fe8a013f650 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593308502 0 0) 0x7fe8a0070480 con 0x7fe8a013f650 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8a0071c70 con 0x7fe8a013f650 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3705152731 0 0) 0x7fe884004eb0 con 0x7fe8a00a9d70 2024-03-29T10:36:52.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8a0070480 con 0x7fe8a00a9d70 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe8900018a0 con 0x7fe8a013f650 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 10 monclient(hunting): dump: 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:52.151 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe89c001750 con 0x7fe8a00a9d70 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 10 monclient(hunting): dump: 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:52.152 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.157+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 975725102 0 0) 0x7fe8a0070480 con 0x7fe8a00a9d70 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 >> v1:172.21.15.136:6789/0 conn(0x7fe8a013f650 legacy=0x7fe8a0139700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 >> v1:172.21.15.117:6789/0 conn(0x7fe8a01457f0 legacy=0x7fe8a0139e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 1 monclient: found mon.c 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a00700d0 con 0x7fe8a00a9d70 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 10 monclient: _finish_auth 0 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe8a86a8640 5 monclient: authenticate success, global_id 4343 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe8a86a8640 10 monclient: _renew_subs 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe8a86a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:52.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.158+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a006f610 con 0x7fe8a00a9d70 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe8a86a8640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe8a86a8640 10 monclient: _renew_subs 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe8a86a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a01df410 con 0x7fe8a00a9d70 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe8a86a8640 1 librados: init done 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe89c001c60 con 0x7fe8a00a9d70 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe896ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe89c004a00 con 0x7fe8a00a9d70 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe896ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.159+0000 7fe896ffd640 10 monclient: dump: 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:52.154 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.160+0000 7fe8a4c41640 4 set_mon_vals no callback set 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.161+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fe89c01a9d0 con 0x7fe8a00a9d70 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.161+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fe89c084800 con 0x7fe8a00a9d70 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.161+0000 7fe896ffd640 10 client.4343.objecter ms_dispatch 0x7fe8a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.161+0000 7fe896ffd640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.161+0000 7fe896ffd640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.161+0000 7fe8a5c43640 10 monclient: get_auth_request con 0x7fe88406df50 auth_method 0 2024-03-29T10:36:52.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.161+0000 7fe896ffd640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T10:36:52.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.161+0000 7fe8a86a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:52.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.162+0000 7fe8a86a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:52.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.162+0000 7fe8a86a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:52.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.162+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe888005080 con 0x7fe8a00a9d70 2024-03-29T10:36:52.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.162+0000 7fe896ffd640 10 client.4343.objecter ms_handle_connect 0x7fe88406df50 2024-03-29T10:36:52.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.169+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe89c04ff70 con 0x7fe8a00a9d70 2024-03-29T10:36:52.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.169+0000 7fe896ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:52.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.169+0000 7fe896ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:52.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.474+0000 7fe8a86a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:52.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.474+0000 7fe8a86a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:52.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.474+0000 7fe8a86a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:36:52.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.474+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe888005330 con 0x7fe8a00a9d70 2024-03-29T10:36:52.462 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:36:52.462 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.475+0000 7fe896ffd640 1 -- 172.21.15.117:0/2521194530 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fe89c083eb0 con 0x7fe8a00a9d70 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.475+0000 7fe896ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.475+0000 7fe896ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.478+0000 7fe8a86a8640 10 librados: watch_flush enter 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.478+0000 7fe8a86a8640 10 librados: watch_flush exit 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.478+0000 7fe8a86a8640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.478+0000 7fe8a86a8640 10 client.4343.objecter successfully canceled tick 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.478+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 >> v1:172.21.15.117:6816/376768171 conn(0x7fe88406df50 legacy=0x7fe884070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.478+0000 7fe8a86a8640 10 monclient: shutdown 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.478+0000 7fe8a86a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:52.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.478+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 >> v1:172.21.15.117:6790/0 conn(0x7fe8a00a9d70 legacy=0x7fe8a01deb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.479+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 shutdown_connections 2024-03-29T10:36:52.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.479+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 >> 172.21.15.117:0/2521194530 conn(0x7fe8a0066a00 msgr2=0x7fe8a0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:52.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.479+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 shutdown_connections 2024-03-29T10:36:52.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.479+0000 7fe8a86a8640 1 -- 172.21.15.117:0/2521194530 wait complete. 2024-03-29T10:36:52.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:52.479+0000 7fe8a86a8640 1 librados: shutdown 2024-03-29T10:36:52.481 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:59.288 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:59.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.449+0000 7f3530c99640 10 monclient: get_monmap_and_config 2024-03-29T10:36:59.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.449+0000 7f3530c99640 10 monclient: build_initial_monmap 2024-03-29T10:36:59.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.449+0000 7f3530c99640 10 monclient: monmap: 2024-03-29T10:36:59.434 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:59.434 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:59.434 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:36:59.450286+0000 2024-03-29T10:36:59.434 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:36:59.450286+0000 2024-03-29T10:36:59.434 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:59.434 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:59.435 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:59.435 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:36:59.435 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:36:59.435 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:59.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.451+0000 7f3530c99640 1 Processor -- start 2024-03-29T10:36:59.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.451+0000 7f3530c99640 1 -- start start 2024-03-29T10:36:59.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.451+0000 7f3530c99640 10 monclient: init 2024-03-29T10:36:59.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 10 monclient(hunting): picked mon.noname-b con 0x7f352c1433e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 10 monclient(hunting): picked mon.noname-a con 0x7f352c0abbf0 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 10 monclient(hunting): picked mon.noname-c con 0x7f352c143e00 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c0adc80 con 0x7f352c0abbf0 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c0aded0 con 0x7f352c143e00 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c0ac110 con 0x7f352c1433e0 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f3530c99640 10 monclient(hunting): authenticate will time out at 940.714479s 2024-03-29T10:36:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f352bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f352c143e00 0x7f352c14b510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:37626/0 (socket says 172.21.15.117:37626) 2024-03-29T10:36:59.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.452+0000 7f352bfff640 1 -- 172.21.15.117:0/1499619808 learned_addr learned my addr 172.21.15.117:0/1499619808 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:59.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.453+0000 7f352a7fc640 1 -- 172.21.15.117:0/1499619808 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294354555 0 0) 0x7f352c0aded0 con 0x7f352c143e00 2024-03-29T10:36:59.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.453+0000 7f352a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:59.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.453+0000 7f352a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:59.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.453+0000 7f352a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:59.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.453+0000 7f352a7fc640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:36:59.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.453+0000 7f352a7fc640 1 -- 172.21.15.117:0/1499619808 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3514003520 con 0x7f352c143e00 2024-03-29T10:36:59.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.454+0000 7f352a7fc640 1 -- 172.21.15.117:0/1499619808 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 892520742 0 0) 0x7f3514003520 con 0x7f352c143e00 2024-03-29T10:36:59.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.454+0000 7f352a7fc640 1 -- 172.21.15.117:0/1499619808 >> v1:172.21.15.117:6790/0 conn(0x7f352c1433e0 legacy=0x7f352c0ac630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:36:59.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 1 -- 172.21.15.117:0/1499619808 >> v1:172.21.15.117:6789/0 conn(0x7f352c0abbf0 legacy=0x7f352c072120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:36:59.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:36:59.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:59.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:36:59.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 1 -- 172.21.15.117:0/1499619808 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c0acd40 con 0x7f352c143e00 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 1 -- 172.21.15.117:0/1499619808 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3524001a10 con 0x7f352c143e00 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f352a7fc640 10 monclient: dump: 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:59.444 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:59.445 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:59.445 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:59.445 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:59.445 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f3530c99640 5 monclient: authenticate success, global_id 4315 2024-03-29T10:36:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.457+0000 7f3530c99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.461+0000 7f352a7fc640 1 -- 172.21.15.117:0/1499619808 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3524001c70 con 0x7f352c143e00 2024-03-29T10:36:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.461+0000 7f352a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.461+0000 7f352a7fc640 1 -- 172.21.15.117:0/1499619808 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3524002080 con 0x7f352c143e00 2024-03-29T10:36:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.461+0000 7f352a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.461+0000 7f352a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.461+0000 7f352a7fc640 10 monclient: dump: 2024-03-29T10:36:59.446 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:59.446 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:59.446 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:59.447 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:59.447 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:59.447 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:59.447 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:59.447 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:59.447 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:59.447 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:59.447 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.461+0000 7f3530c99640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:59.447 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.461+0000 7f3530c99640 4 set_mon_vals no callback set 2024-03-29T10:36:59.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.462+0000 7f3530c99640 10 monclient: shutdown 2024-03-29T10:36:59.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.462+0000 7f3530c99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:59.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.462+0000 7f3530c99640 1 -- 172.21.15.117:0/1499619808 >> v1:172.21.15.136:6789/0 conn(0x7f352c143e00 legacy=0x7f352c14b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.462+0000 7f3530c99640 1 -- 172.21.15.117:0/1499619808 shutdown_connections 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.462+0000 7f3530c99640 1 -- 172.21.15.117:0/1499619808 >> 172.21.15.117:0/1499619808 conn(0x7f352c0b3c50 msgr2=0x7f352c0b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.463+0000 7f3530c99640 1 -- 172.21.15.117:0/1499619808 shutdown_connections 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.463+0000 7f3530c99640 1 -- 172.21.15.117:0/1499619808 wait complete. 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.464+0000 7f3530c99640 10 monclient: build_initial_monmap 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.464+0000 7f3530c99640 10 monclient: monmap: 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:59.449 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:59.450 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:36:59.450 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:36:59.450 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:36:59.450 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:59.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.464+0000 7f3530c99640 1 librados: starting msgr at 2024-03-29T10:36:59.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.464+0000 7f3530c99640 1 librados: starting objecter 2024-03-29T10:36:59.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.465+0000 7f3530c99640 1 Processor -- start 2024-03-29T10:36:59.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.465+0000 7f3530c99640 1 -- start start 2024-03-29T10:36:59.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.465+0000 7f3530c99640 1 librados: setting wanted keys 2024-03-29T10:36:59.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.465+0000 7f3530c99640 1 librados: calling monclient init 2024-03-29T10:36:59.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.465+0000 7f3530c99640 10 monclient: init 2024-03-29T10:36:59.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.466+0000 7f3530c99640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:59.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.468+0000 7f3530c99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:59.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.468+0000 7f3530c99640 10 monclient(hunting): picked mon.noname-c con 0x7f352c0abbf0 addr v1:172.21.15.117:6790/0 2024-03-29T10:36:59.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.468+0000 7f3530c99640 10 monclient(hunting): picked mon.noname-a con 0x7f352c1433e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:36:59.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.468+0000 7f3530c99640 10 monclient(hunting): picked mon.noname-b con 0x7f352c143e00 addr v1:172.21.15.136:6789/0 2024-03-29T10:36:59.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.468+0000 7f3530c99640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c139500 con 0x7f352c1433e0 2024-03-29T10:36:59.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.468+0000 7f3530c99640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c137ac0 con 0x7f352c143e00 2024-03-29T10:36:59.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.468+0000 7f3530c99640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352c1377b0 con 0x7f352c0abbf0 2024-03-29T10:36:59.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.468+0000 7f3530c99640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:59.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.468+0000 7f3530c99640 10 monclient(hunting): authenticate will time out at 940.730537s 2024-03-29T10:36:59.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.469+0000 7f352bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f352c143e00 0x7f352c1df4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:41100/0 (socket says 172.21.15.117:41100) 2024-03-29T10:36:59.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.469+0000 7f352bfff640 1 -- 172.21.15.117:0/3469086657 learned_addr learned my addr 172.21.15.117:0/3469086657 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:36:59.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.470+0000 7f3513fff640 10 client.?.objecter ms_handle_connect 0x7f352c143e00 2024-03-29T10:36:59.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.470+0000 7f3513fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:59.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.470+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510276580 0 0) 0x7f352c137ac0 con 0x7f352c143e00 2024-03-29T10:36:59.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.470+0000 7f3513fff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:59.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.470+0000 7f3513fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:59.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.470+0000 7f3513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:59.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.470+0000 7f3513fff640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:36:59.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.470+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3518004eb0 con 0x7f352c143e00 2024-03-29T10:36:59.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.471+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1354587432 0 0) 0x7f3518004eb0 con 0x7f352c143e00 2024-03-29T10:36:59.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.471+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f352c137ac0 con 0x7f352c143e00 2024-03-29T10:36:59.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.471+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f35240019d0 con 0x7f352c143e00 2024-03-29T10:36:59.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.471+0000 7f3513fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:59.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.471+0000 7f3513fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:59.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.471+0000 7f3513fff640 10 monclient(hunting): dump: 2024-03-29T10:36:59.457 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:59.457 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:59.457 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:59.457 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:59.458 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:59.458 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:59.458 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:59.458 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:59.458 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:59.458 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:59.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.471+0000 7f3513fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:59.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.472+0000 7f3513fff640 10 client.?.objecter ms_handle_connect 0x7f352c0abbf0 2024-03-29T10:36:59.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.472+0000 7f3513fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.472+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506513661 0 0) 0x7f352c1377b0 con 0x7f352c0abbf0 2024-03-29T10:36:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:36:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3518004eb0 con 0x7f352c0abbf0 2024-03-29T10:36:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1910773677 0 0) 0x7f352c137ac0 con 0x7f352c143e00 2024-03-29T10:36:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 >> v1:172.21.15.117:6790/0 conn(0x7f352c0abbf0 legacy=0x7f352c1370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 >> v1:172.21.15.117:6789/0 conn(0x7f352c1433e0 legacy=0x7f352c1d4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 10 monclient: _finish_hunting 0 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 1 monclient: found mon.b 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.473+0000 7f3513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3513fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c139b90 con 0x7f352c143e00 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3513fff640 10 monclient: _finish_auth 0 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3513fff640 10 monclient: _check_auth_tickets 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3513fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3530c99640 5 monclient: authenticate success, global_id 4318 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3530c99640 10 monclient: _renew_subs 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3530c99640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352c139d80 con 0x7f352c143e00 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3530c99640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3530c99640 10 monclient: _renew_subs 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3530c99640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f352c1d5480 con 0x7f352c143e00 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3530c99640 1 librados: init done 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3524004e10 con 0x7f352c143e00 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3513fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.474+0000 7f3529ffb640 4 set_mon_vals no callback set 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.475+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f35240051a0 con 0x7f352c143e00 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.475+0000 7f3513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.475+0000 7f3513fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.475+0000 7f3513fff640 10 monclient: dump: 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:36:59.467 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.476+0000 7f3530c99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.476+0000 7f3530c99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.476+0000 7f3530c99640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.476+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f3524005380 con 0x7f352c143e00 2024-03-29T10:36:59.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.476+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c005080 con 0x7f352c143e00 2024-03-29T10:36:59.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.477+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f3524085050 con 0x7f352c143e00 2024-03-29T10:36:59.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.478+0000 7f3513fff640 10 client.4318.objecter ms_dispatch 0x7f352c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:59.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.478+0000 7f3513fff640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:59.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.478+0000 7f3513fff640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:59.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.478+0000 7f352b7fe640 10 monclient: get_auth_request con 0x7f351806db10 auth_method 0 2024-03-29T10:36:59.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.478+0000 7f3513fff640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-29T10:36:59.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.478+0000 7f3513fff640 10 client.4318.objecter ms_handle_connect 0x7f351806db10 2024-03-29T10:36:59.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.486+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3524050930 con 0x7f352c143e00 2024-03-29T10:36:59.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.486+0000 7f3513fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:59.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.486+0000 7f3513fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:59.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.804+0000 7f3530c99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:59.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.805+0000 7f3530c99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:59.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.805+0000 7f3530c99640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:36:59.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.805+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f350c005330 con 0x7f352c143e00 2024-03-29T10:36:59.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.805+0000 7f3513fff640 1 -- 172.21.15.117:0/3469086657 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f3524052860 con 0x7f352c143e00 2024-03-29T10:36:59.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.805+0000 7f3513fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:59.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.805+0000 7f3513fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:59.791 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:36:59.791 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:59.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 10 librados: watch_flush enter 2024-03-29T10:36:59.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 10 librados: watch_flush exit 2024-03-29T10:36:59.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-29T10:36:59.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 10 client.4318.objecter successfully canceled tick 2024-03-29T10:36:59.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 >> v1:172.21.15.117:6816/376768171 conn(0x7f351806db10 legacy=0x7f351806ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 10 monclient: shutdown 2024-03-29T10:36:59.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:59.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 >> v1:172.21.15.136:6789/0 conn(0x7f352c143e00 legacy=0x7f352c1df4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 shutdown_connections 2024-03-29T10:36:59.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.809+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 >> 172.21.15.117:0/3469086657 conn(0x7f352c0b3c50 msgr2=0x7f352c1352b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:59.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.810+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 shutdown_connections 2024-03-29T10:36:59.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.810+0000 7f3530c99640 1 -- 172.21.15.117:0/3469086657 wait complete. 2024-03-29T10:36:59.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:36:59.810+0000 7f3530c99640 1 librados: shutdown 2024-03-29T10:36:59.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:06.618 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.789+0000 7f531bfff640 10 monclient: get_monmap_and_config 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.789+0000 7f531bfff640 10 monclient: build_initial_monmap 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.789+0000 7f531bfff640 10 monclient: monmap: 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:37:06.790193+0000 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:37:06.790193+0000 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:06.774 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:06.775 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:37:06.775 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:37:06.775 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:06.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.790+0000 7f531bfff640 1 Processor -- start 2024-03-29T10:37:06.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.790+0000 7f531bfff640 1 -- start start 2024-03-29T10:37:06.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.790+0000 7f531bfff640 10 monclient: init 2024-03-29T10:37:06.776 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:06.777 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:37:06.777 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f531c0b2790 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:06.777 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f531c143c80 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:06.777 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f531c146e30 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:06.777 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c146290 con 0x7f531c143c80 2024-03-29T10:37:06.777 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c0b3970 con 0x7f531c146e30 2024-03-29T10:37:06.777 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c06ee80 con 0x7f531c0b2790 2024-03-29T10:37:06.777 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:06.777 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.791+0000 7f531bfff640 10 monclient(hunting): authenticate will time out at 948.053268s 2024-03-29T10:37:06.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.794+0000 7f531a7fc640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f531c143c80 0x7f531c146720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:42937/0 (socket says 172.21.15.117:42937) 2024-03-29T10:37:06.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.794+0000 7f531a7fc640 1 -- 172.21.15.117:0/3862094873 learned_addr learned my addr 172.21.15.117:0/3862094873 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:06.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.794+0000 7f5319ffb640 1 -- 172.21.15.117:0/3862094873 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110714906 0 0) 0x7f531c146290 con 0x7f531c143c80 2024-03-29T10:37:06.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.794+0000 7f5319ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:37:06.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.794+0000 7f5319ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:06.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.794+0000 7f5319ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:06.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.794+0000 7f5319ffb640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:37:06.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.794+0000 7f5319ffb640 1 -- 172.21.15.117:0/3862094873 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5304003520 con 0x7f531c143c80 2024-03-29T10:37:06.786 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 1 -- 172.21.15.117:0/3862094873 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4259129599 0 0) 0x7f5304003520 con 0x7f531c143c80 2024-03-29T10:37:06.786 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 1 -- 172.21.15.117:0/3862094873 >> v1:172.21.15.117:6790/0 conn(0x7f531c0b2790 legacy=0x7f531c143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.786 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 1 -- 172.21.15.117:0/3862094873 >> v1:172.21.15.136:6789/0 conn(0x7f531c146e30 legacy=0x7f531c151bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 1 monclient: found mon.noname-a 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 1 -- 172.21.15.117:0/3862094873 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c06f160 con 0x7f531c143c80 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 10 monclient: _finish_auth 0 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 1 -- 172.21.15.117:0/3862094873 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f530c001690 con 0x7f531c143c80 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.799+0000 7f5319ffb640 10 monclient: dump: 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:06.787 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.800+0000 7f5319ffb640 1 -- 172.21.15.117:0/3862094873 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f530c001860 con 0x7f531c143c80 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.800+0000 7f5319ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.800+0000 7f5319ffb640 1 -- 172.21.15.117:0/3862094873 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f530c001cb0 con 0x7f531c143c80 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.800+0000 7f5319ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.800+0000 7f5319ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.800+0000 7f5319ffb640 10 monclient: dump: 2024-03-29T10:37:06.788 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.802+0000 7f531bfff640 5 monclient: authenticate success, global_id 4323 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.802+0000 7f531bfff640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:06.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.802+0000 7f531bfff640 4 set_mon_vals no callback set 2024-03-29T10:37:06.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.803+0000 7f531bfff640 10 monclient: shutdown 2024-03-29T10:37:06.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.803+0000 7f531bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:06.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.803+0000 7f531bfff640 1 -- 172.21.15.117:0/3862094873 >> v1:172.21.15.117:6789/0 conn(0x7f531c143c80 legacy=0x7f531c146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.806+0000 7f531bfff640 1 -- 172.21.15.117:0/3862094873 shutdown_connections 2024-03-29T10:37:06.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.806+0000 7f531bfff640 1 -- 172.21.15.117:0/3862094873 >> 172.21.15.117:0/3862094873 conn(0x7f531c06d070 msgr2=0x7f531c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:06.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.807+0000 7f531bfff640 1 -- 172.21.15.117:0/3862094873 shutdown_connections 2024-03-29T10:37:06.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.807+0000 7f531bfff640 1 -- 172.21.15.117:0/3862094873 wait complete. 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.808+0000 7f531bfff640 10 monclient: build_initial_monmap 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.808+0000 7f531bfff640 10 monclient: monmap: 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:37:06.793 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:06.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.808+0000 7f531bfff640 1 librados: starting msgr at 2024-03-29T10:37:06.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.808+0000 7f531bfff640 1 librados: starting objecter 2024-03-29T10:37:06.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.808+0000 7f531bfff640 1 Processor -- start 2024-03-29T10:37:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.809+0000 7f531bfff640 1 -- start start 2024-03-29T10:37:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.809+0000 7f531bfff640 1 librados: setting wanted keys 2024-03-29T10:37:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.809+0000 7f531bfff640 1 librados: calling monclient init 2024-03-29T10:37:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.809+0000 7f531bfff640 10 monclient: init 2024-03-29T10:37:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.810+0000 7f531bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.810+0000 7f531bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.810+0000 7f531bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f531c0b2790 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.810+0000 7f531bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f531c143c80 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.812+0000 7f531bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f531c146e30 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.812+0000 7f531bfff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c0b77e0 con 0x7f531c146e30 2024-03-29T10:37:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.812+0000 7f531bfff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c142fb0 con 0x7f531c143c80 2024-03-29T10:37:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.812+0000 7f531bfff640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531c1399a0 con 0x7f531c0b2790 2024-03-29T10:37:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.812+0000 7f531bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.812+0000 7f531bfff640 10 monclient(hunting): authenticate will time out at 948.074303s 2024-03-29T10:37:06.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.816+0000 7f531a7fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f531c143c80 0x7f531c1e46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:55427/0 (socket says 172.21.15.117:55427) 2024-03-29T10:37:06.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.816+0000 7f531a7fc640 1 -- 172.21.15.117:0/1179596425 learned_addr learned my addr 172.21.15.117:0/1179596425 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 10 client.?.objecter ms_handle_connect 0x7f531c143c80 2024-03-29T10:37:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279532071 0 0) 0x7f531c142fb0 con 0x7f531c143c80 2024-03-29T10:37:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:06.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:06.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:37:06.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5308004eb0 con 0x7f531c143c80 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2645471991 0 0) 0x7f5308004eb0 con 0x7f531c143c80 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.817+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f531c142fb0 con 0x7f531c143c80 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f530c001b70 con 0x7f531c143c80 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 10 monclient(hunting): dump: 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:06.805 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3167842199 0 0) 0x7f531c142fb0 con 0x7f531c143c80 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 >> v1:172.21.15.117:6790/0 conn(0x7f531c0b2790 legacy=0x7f531c1e3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 >> v1:172.21.15.117:6789/0 conn(0x7f531c146e30 legacy=0x7f531c1ee7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 10 monclient: _finish_hunting 0 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 1 monclient: found mon.b 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:06.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c139b30 con 0x7f531c143c80 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 10 monclient: _finish_auth 0 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 10 monclient: _check_auth_tickets 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f53037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f531bfff640 5 monclient: authenticate success, global_id 4321 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f531bfff640 10 monclient: _renew_subs 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f531bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c145450 con 0x7f531c143c80 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f531bfff640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f531bfff640 10 monclient: _renew_subs 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.818+0000 7f531bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.819+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c0b7130 con 0x7f531c143c80 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.819+0000 7f531bfff640 1 librados: init done 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.819+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f530c004bf0 con 0x7f531c143c80 2024-03-29T10:37:06.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.819+0000 7f53037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.819+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f530c004f00 con 0x7f531c143c80 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.819+0000 7f53037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.819+0000 7f53037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.819+0000 7f53037fe640 10 monclient: dump: 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:06.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.819+0000 7f53197fa640 4 set_mon_vals no callback set 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.821+0000 7f531bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.821+0000 7f531bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.821+0000 7f531bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.821+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52fc005080 con 0x7f531c143c80 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.822+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f530c01aed0 con 0x7f531c143c80 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.822+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f530c084c30 con 0x7f531c143c80 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.822+0000 7f53037fe640 10 client.4321.objecter ms_dispatch 0x7f531c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.822+0000 7f53037fe640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.822+0000 7f53037fe640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.823+0000 7f53037fe640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.823+0000 7f531affd640 10 monclient: get_auth_request con 0x7f530806dc70 auth_method 0 2024-03-29T10:37:06.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.823+0000 7f53037fe640 10 client.4321.objecter ms_handle_connect 0x7f530806dc70 2024-03-29T10:37:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.831+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f530c050510 con 0x7f531c143c80 2024-03-29T10:37:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.831+0000 7f53037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:06.831+0000 7f53037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:07.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.158+0000 7f531bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:07.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.158+0000 7f531bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:07.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.158+0000 7f531bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:07.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.158+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52fc005330 con 0x7f531c143c80 2024-03-29T10:37:07.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.163+0000 7f53037fe640 1 -- 172.21.15.117:0/1179596425 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f530c052440 con 0x7f531c143c80 2024-03-29T10:37:07.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.163+0000 7f53037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:07.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.163+0000 7f53037fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:07.149 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:37:07.149 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 10 librados: watch_flush enter 2024-03-29T10:37:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 10 librados: watch_flush exit 2024-03-29T10:37:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-29T10:37:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 10 client.4321.objecter successfully canceled tick 2024-03-29T10:37:07.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 >> v1:172.21.15.117:6816/376768171 conn(0x7f530806dc70 legacy=0x7f5308070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:07.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 10 monclient: shutdown 2024-03-29T10:37:07.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:07.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 >> v1:172.21.15.136:6789/0 conn(0x7f531c143c80 legacy=0x7f531c1e46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:07.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 shutdown_connections 2024-03-29T10:37:07.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.168+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 >> 172.21.15.117:0/1179596425 conn(0x7f531c06d070 msgr2=0x7f531c0b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:07.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.169+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 shutdown_connections 2024-03-29T10:37:07.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.169+0000 7f531bfff640 1 -- 172.21.15.117:0/1179596425 wait complete. 2024-03-29T10:37:07.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:07.169+0000 7f531bfff640 1 librados: shutdown 2024-03-29T10:37:07.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:13.978 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.141+0000 7f466bfff640 10 monclient: get_monmap_and_config 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.141+0000 7f466bfff640 10 monclient: build_initial_monmap 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.141+0000 7f466bfff640 10 monclient: monmap: 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:37:14.142508+0000 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:37:14.142508+0000 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:14.126 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:14.127 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:37:14.127 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:37:14.127 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:14.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.143+0000 7f466bfff640 1 Processor -- start 2024-03-29T10:37:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.143+0000 7f466bfff640 1 -- start start 2024-03-29T10:37:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.143+0000 7f466bfff640 10 monclient: init 2024-03-29T10:37:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:14.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:14.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f466c0b2800 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:14.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f466c0b3220 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:14.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f466c1439c0 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:14.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466c0b3840 con 0x7f466c0b2800 2024-03-29T10:37:14.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466c06f230 con 0x7f466c1439c0 2024-03-29T10:37:14.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466c06ef50 con 0x7f466c0b3220 2024-03-29T10:37:14.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:14.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.144+0000 7f466bfff640 10 monclient(hunting): authenticate will time out at 955.406442s 2024-03-29T10:37:14.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.150+0000 7f466b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f466c1439c0 0x7f466c151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:50139/0 (socket says 172.21.15.117:50139) 2024-03-29T10:37:14.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.150+0000 7f466b7fe640 1 -- 172.21.15.117:0/3254944730 learned_addr learned my addr 172.21.15.117:0/3254944730 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:14.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.151+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305916560 0 0) 0x7f466c0b3840 con 0x7f466c0b2800 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.151+0000 7f4669ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.151+0000 7f4669ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.151+0000 7f4669ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.151+0000 7f4669ffb640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.151+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4654003520 con 0x7f466c0b2800 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384559154 0 0) 0x7f466c06f230 con 0x7f466c1439c0 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f466c0b3840 con 0x7f466c1439c0 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1111517720 0 0) 0x7f4654003520 con 0x7f466c0b2800 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 >> v1:172.21.15.117:6790/0 conn(0x7f466c0b3220 legacy=0x7f466c146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:37:14.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 >> v1:172.21.15.136:6789/0 conn(0x7f466c1439c0 legacy=0x7f466c151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 monclient: found mon.noname-a 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c071d30 con 0x7f466c0b2800 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: _finish_auth 0 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f465c001690 con 0x7f466c0b2800 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: dump: 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:14.138 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f465c0017f0 con 0x7f466c0b2800 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 1 -- 172.21.15.117:0/3254944730 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f465c001c00 con 0x7f466c0b2800 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f4669ffb640 10 monclient: dump: 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f466bfff640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f466bfff640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:14.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.152+0000 7f466bfff640 4 set_mon_vals no callback set 2024-03-29T10:37:14.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.153+0000 7f466bfff640 10 monclient: shutdown 2024-03-29T10:37:14.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.153+0000 7f466bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:14.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.153+0000 7f466bfff640 1 -- 172.21.15.117:0/3254944730 >> v1:172.21.15.117:6789/0 conn(0x7f466c0b2800 legacy=0x7f466c1432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:14.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.153+0000 7f466bfff640 1 -- 172.21.15.117:0/3254944730 shutdown_connections 2024-03-29T10:37:14.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.154+0000 7f466bfff640 1 -- 172.21.15.117:0/3254944730 >> 172.21.15.117:0/3254944730 conn(0x7f466c066a00 msgr2=0x7f466c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:14.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.154+0000 7f466bfff640 1 -- 172.21.15.117:0/3254944730 shutdown_connections 2024-03-29T10:37:14.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.154+0000 7f466bfff640 1 -- 172.21.15.117:0/3254944730 wait complete. 2024-03-29T10:37:14.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.155+0000 7f466bfff640 10 monclient: build_initial_monmap 2024-03-29T10:37:14.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.155+0000 7f466bfff640 10 monclient: monmap: 2024-03-29T10:37:14.144 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:14.144 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.155+0000 7f466bfff640 1 librados: starting msgr at 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.155+0000 7f466bfff640 1 librados: starting objecter 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.156+0000 7f466bfff640 1 Processor -- start 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.156+0000 7f466bfff640 1 -- start start 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.156+0000 7f466bfff640 1 librados: setting wanted keys 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.156+0000 7f466bfff640 1 librados: calling monclient init 2024-03-29T10:37:14.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.156+0000 7f466bfff640 10 monclient: init 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f466c0b2800 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f466c0b3220 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f466c1439c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466c148360 con 0x7f466c1439c0 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466c149e60 con 0x7f466c0b3220 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466c1490a0 con 0x7f466c0b2800 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:14.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.157+0000 7f466bfff640 10 monclient(hunting): authenticate will time out at 955.419588s 2024-03-29T10:37:14.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.165+0000 7f466b7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f466c1439c0 0x7f466c1de890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:46623/0 (socket says 172.21.15.117:46623) 2024-03-29T10:37:14.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.165+0000 7f466affd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f466c0b2800 0x7f466c148990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:37019/0 (socket says 172.21.15.117:37019) 2024-03-29T10:37:14.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.165+0000 7f466b7fe640 1 -- 172.21.15.117:0/320916052 learned_addr learned my addr 172.21.15.117:0/320916052 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.165+0000 7f46537fe640 10 client.?.objecter ms_handle_connect 0x7f466c0b2800 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.165+0000 7f46537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.165+0000 7f46537fe640 10 client.?.objecter ms_handle_connect 0x7f466c0b3220 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.165+0000 7f46537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.165+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359060297 0 0) 0x7f466c149e60 con 0x7f466c0b3220 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.165+0000 7f46537fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.166+0000 7f46537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.166+0000 7f46537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.166+0000 7f46537fe640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.166+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4658004ee0 con 0x7f466c0b3220 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.166+0000 7f46537fe640 10 client.?.objecter ms_handle_connect 0x7f466c1439c0 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.166+0000 7f46537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:14.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.166+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328475439 0 0) 0x7f466c1490a0 con 0x7f466c0b2800 2024-03-29T10:37:14.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f466c149e60 con 0x7f466c0b2800 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2113419488 0 0) 0x7f4658004ee0 con 0x7f466c0b3220 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f466c1490a0 con 0x7f466c0b3220 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4664001890 con 0x7f466c0b3220 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.168+0000 7f46537fe640 10 monclient(hunting): dump: 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.153 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.154 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:14.154 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:14.154 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:14.154 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:14.154 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:14.154 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.169+0000 7f46537fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.170+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049674118 0 0) 0x7f466c148360 con 0x7f466c1439c0 2024-03-29T10:37:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.170+0000 7f46537fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.170+0000 7f46537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.170+0000 7f46537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.170+0000 7f46537fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:37:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.170+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4658004ee0 con 0x7f466c1439c0 2024-03-29T10:37:14.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.172+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 425800762 0 0) 0x7f466c149e60 con 0x7f466c0b2800 2024-03-29T10:37:14.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.172+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f466c148360 con 0x7f466c0b2800 2024-03-29T10:37:14.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.172+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1369712915 0 0) 0x7f4658004ee0 con 0x7f466c1439c0 2024-03-29T10:37:14.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f466c149e60 con 0x7f466c1439c0 2024-03-29T10:37:14.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f465c001b00 con 0x7f466c0b2800 2024-03-29T10:37:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 10 monclient(hunting): dump: 2024-03-29T10:37:14.159 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:14.159 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:14.159 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.159 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.159 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:14.159 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1700817968 0 0) 0x7f466c1490a0 con 0x7f466c0b3220 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 >> v1:172.21.15.117:6790/0 conn(0x7f466c0b2800 legacy=0x7f466c148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 >> v1:172.21.15.117:6789/0 conn(0x7f466c1439c0 legacy=0x7f466c1de890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 10 monclient: _finish_hunting 0 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 1 monclient: found mon.b 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c147a70 con 0x7f466c0b3220 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 10 monclient: _finish_auth 0 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 10 monclient: _check_auth_tickets 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f46537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f466bfff640 5 monclient: authenticate success, global_id 4327 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f466bfff640 10 monclient: _renew_subs 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f466bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c13c880 con 0x7f466c0b3220 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f466bfff640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f466bfff640 10 monclient: _renew_subs 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f466bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c13c450 con 0x7f466c0b3220 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.173+0000 7f466bfff640 1 librados: init done 2024-03-29T10:37:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.174+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4664001da0 con 0x7f466c0b3220 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.174+0000 7f46537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.174+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4664003c90 con 0x7f466c0b3220 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.174+0000 7f46537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.174+0000 7f46537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.174+0000 7f46537fe640 10 monclient: dump: 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:14.162 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:14.163 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:14.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.175+0000 7f4668ff9640 4 set_mon_vals no callback set 2024-03-29T10:37:14.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.177+0000 7f466bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:14.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.177+0000 7f466bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:14.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.177+0000 7f466bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:14.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.177+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c005080 con 0x7f466c0b3220 2024-03-29T10:37:14.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.178+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f4664019c60 con 0x7f466c0b3220 2024-03-29T10:37:14.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.180+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f4664083b10 con 0x7f466c0b3220 2024-03-29T10:37:14.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.180+0000 7f46537fe640 10 client.4327.objecter ms_dispatch 0x7f466c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:14.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.180+0000 7f46537fe640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:14.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.180+0000 7f46537fe640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:14.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.180+0000 7f46537fe640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T10:37:14.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.181+0000 7f466affd640 10 monclient: get_auth_request con 0x7f465806e100 auth_method 0 2024-03-29T10:37:14.166 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.182+0000 7f46537fe640 10 client.4327.objecter ms_handle_connect 0x7f465806e100 2024-03-29T10:37:14.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.187+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f466404f3f0 con 0x7f466c0b3220 2024-03-29T10:37:14.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.188+0000 7f46537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:14.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.188+0000 7f46537fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:14.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.520+0000 7f466bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:14.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.520+0000 7f466bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:14.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.520+0000 7f466bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:14.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.520+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f464c005330 con 0x7f466c0b3220 2024-03-29T10:37:14.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.524+0000 7f46537fe640 1 -- 172.21.15.117:0/320916052 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f4664051320 con 0x7f466c0b3220 2024-03-29T10:37:14.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.524+0000 7f46537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:14.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.524+0000 7f46537fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:14.509 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:37:14.509 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:14.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.528+0000 7f466bfff640 10 librados: watch_flush enter 2024-03-29T10:37:14.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 10 librados: watch_flush exit 2024-03-29T10:37:14.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T10:37:14.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 10 client.4327.objecter successfully canceled tick 2024-03-29T10:37:14.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 >> v1:172.21.15.117:6816/376768171 conn(0x7f465806e100 legacy=0x7f46580705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:14.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 10 monclient: shutdown 2024-03-29T10:37:14.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:14.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 >> v1:172.21.15.136:6789/0 conn(0x7f466c0b3220 legacy=0x7f466c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:14.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 shutdown_connections 2024-03-29T10:37:14.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 >> 172.21.15.117:0/320916052 conn(0x7f466c066a00 msgr2=0x7f466c141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:14.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 shutdown_connections 2024-03-29T10:37:14.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.531+0000 7f466bfff640 1 -- 172.21.15.117:0/320916052 wait complete. 2024-03-29T10:37:14.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:14.532+0000 7f466bfff640 1 librados: shutdown 2024-03-29T10:37:14.534 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:21.341 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:21.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.509+0000 7f4c3b1cd640 10 monclient: get_monmap_and_config 2024-03-29T10:37:21.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.509+0000 7f4c3b1cd640 10 monclient: build_initial_monmap 2024-03-29T10:37:21.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.510+0000 7f4c3b1cd640 10 monclient: monmap: 2024-03-29T10:37:21.494 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:21.494 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:21.494 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:37:21.510920+0000 2024-03-29T10:37:21.495 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:37:21.510920+0000 2024-03-29T10:37:21.495 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:21.495 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:21.495 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:21.495 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:37:21.495 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:37:21.495 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:21.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.511+0000 7f4c3b1cd640 1 Processor -- start 2024-03-29T10:37:21.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.511+0000 7f4c3b1cd640 1 -- start start 2024-03-29T10:37:21.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.511+0000 7f4c3b1cd640 10 monclient: init 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 10 monclient(hunting): picked mon.noname-c con 0x7f4c3406ab70 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 10 monclient(hunting): picked mon.noname-b con 0x7f4c3406af50 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 10 monclient(hunting): picked mon.noname-a con 0x7f4c34141fb0 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340b0d50 con 0x7f4c34141fb0 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340b1030 con 0x7f4c3406ab70 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340afa70 con 0x7f4c3406af50 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.512+0000 7f4c3b1cd640 10 monclient(hunting): authenticate will time out at 962.774873s 2024-03-29T10:37:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c3976a640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f4c34141fb0 0x7f4c340715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:56609/0 (socket says 172.21.15.117:56609) 2024-03-29T10:37:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c3976a640 1 -- 172.21.15.117:0/2830763070 learned_addr learned my addr 172.21.15.117:0/2830763070 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c2b7fe640 1 -- 172.21.15.117:0/2830763070 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906895555 0 0) 0x7f4c340b0d50 con 0x7f4c34141fb0 2024-03-29T10:37:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c2b7fe640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c2b7fe640 1 -- 172.21.15.117:0/2830763070 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c1c003520 con 0x7f4c34141fb0 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c2b7fe640 1 -- 172.21.15.117:0/2830763070 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063862079 0 0) 0x7f4c1c003520 con 0x7f4c34141fb0 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c2b7fe640 1 -- 172.21.15.117:0/2830763070 >> v1:172.21.15.117:6790/0 conn(0x7f4c3406af50 legacy=0x7f4c34072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.518+0000 7f4c2b7fe640 1 -- 172.21.15.117:0/2830763070 >> v1:172.21.15.136:6789/0 conn(0x7f4c3406ab70 legacy=0x7f4c340af2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.519+0000 7f4c2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.519+0000 7f4c2b7fe640 1 monclient: found mon.noname-a 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.519+0000 7f4c2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.519+0000 7f4c2b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.519+0000 7f4c2b7fe640 1 -- 172.21.15.117:0/2830763070 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c341423e0 con 0x7f4c34141fb0 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.519+0000 7f4c2b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.519+0000 7f4c2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.519+0000 7f4c2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.519+0000 7f4c2b7fe640 1 -- 172.21.15.117:0/2830763070 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4c2c001a10 con 0x7f4c34141fb0 2024-03-29T10:37:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.520+0000 7f4c2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.520+0000 7f4c2b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.520+0000 7f4c2b7fe640 10 monclient: dump: 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:21.506 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:21.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.523+0000 7f4c3b1cd640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:37:21.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.523+0000 7f4c3b1cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:21.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.523+0000 7f4c2b7fe640 1 -- 172.21.15.117:0/2830763070 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c2c001ce0 con 0x7f4c34141fb0 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.523+0000 7f4c2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.523+0000 7f4c3b1cd640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.523+0000 7f4c3b1cd640 4 set_mon_vals no callback set 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.523+0000 7f4c2b7fe640 1 -- 172.21.15.117:0/2830763070 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4c2c002070 con 0x7f4c34141fb0 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.524+0000 7f4c2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.524+0000 7f4c2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.524+0000 7f4c2b7fe640 10 monclient: dump: 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:21.510 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:21.510 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:21.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.524+0000 7f4c3b1cd640 10 monclient: shutdown 2024-03-29T10:37:21.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.524+0000 7f4c3b1cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:21.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.524+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2830763070 >> v1:172.21.15.117:6789/0 conn(0x7f4c34141fb0 legacy=0x7f4c340715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.526+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2830763070 shutdown_connections 2024-03-29T10:37:21.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.526+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2830763070 >> 172.21.15.117:0/2830763070 conn(0x7f4c34067470 msgr2=0x7f4c340b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:21.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.527+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2830763070 shutdown_connections 2024-03-29T10:37:21.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.527+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2830763070 wait complete. 2024-03-29T10:37:21.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.528+0000 7f4c3b1cd640 10 monclient: build_initial_monmap 2024-03-29T10:37:21.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.528+0000 7f4c3b1cd640 10 monclient: monmap: 2024-03-29T10:37:21.513 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:21.513 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:21.513 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:37:21.513 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:37:21.513 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.528+0000 7f4c3b1cd640 1 librados: starting msgr at 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.528+0000 7f4c3b1cd640 1 librados: starting objecter 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.528+0000 7f4c3b1cd640 1 Processor -- start 2024-03-29T10:37:21.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.529+0000 7f4c3b1cd640 1 -- start start 2024-03-29T10:37:21.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.529+0000 7f4c3b1cd640 1 librados: setting wanted keys 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.529+0000 7f4c3b1cd640 1 librados: calling monclient init 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.529+0000 7f4c3b1cd640 10 monclient: init 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 10 monclient(hunting): picked mon.noname-b con 0x7f4c3406ab70 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 10 monclient(hunting): picked mon.noname-c con 0x7f4c34141fb0 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 10 monclient(hunting): picked mon.noname-a con 0x7f4c341e38b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340ab570 con 0x7f4c341e38b0 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c341e7e90 con 0x7f4c3406ab70 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c34135740 con 0x7f4c34141fb0 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.530+0000 7f4c3b1cd640 10 monclient(hunting): authenticate will time out at 962.792196s 2024-03-29T10:37:21.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.536+0000 7f4c38f69640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4c3406ab70 0x7f4c341e31a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:60849/0 (socket says 172.21.15.117:60849) 2024-03-29T10:37:21.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.536+0000 7f4c38f69640 1 -- 172.21.15.117:0/2740004761 learned_addr learned my addr 172.21.15.117:0/2740004761 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:21.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.536+0000 7f4c297fa640 10 client.?.objecter ms_handle_connect 0x7f4c3406ab70 2024-03-29T10:37:21.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.536+0000 7f4c297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:21.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.537+0000 7f4c297fa640 10 client.?.objecter ms_handle_connect 0x7f4c341e38b0 2024-03-29T10:37:21.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.537+0000 7f4c297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:21.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.537+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912161125 0 0) 0x7f4c341e7e90 con 0x7f4c3406ab70 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.537+0000 7f4c297fa640 10 monclient(hunting): _negotiate 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.537+0000 7f4c297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.537+0000 7f4c297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.537+0000 7f4c297fa640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c20004eb0 con 0x7f4c3406ab70 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 10 client.?.objecter ms_handle_connect 0x7f4c34141fb0 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999527087 0 0) 0x7f4c340ab570 con 0x7f4c341e38b0 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 10 monclient(hunting): _negotiate 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c341e7e90 con 0x7f4c341e38b0 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2426039488 0 0) 0x7f4c20004eb0 con 0x7f4c3406ab70 2024-03-29T10:37:21.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c340ab570 con 0x7f4c3406ab70 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4c30001870 con 0x7f4c3406ab70 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.538+0000 7f4c297fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.539+0000 7f4c297fa640 10 monclient(hunting): dump: 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:21.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.539+0000 7f4c297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:21.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 689605797 0 0) 0x7f4c340ab570 con 0x7f4c3406ab70 2024-03-29T10:37:21.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 >> v1:172.21.15.117:6790/0 conn(0x7f4c34141fb0 legacy=0x7f4c341e3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 >> v1:172.21.15.117:6789/0 conn(0x7f4c341e38b0 legacy=0x7f4c341e4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 10 monclient: _finish_hunting 0 2024-03-29T10:37:21.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 1 monclient: found mon.b 2024-03-29T10:37:21.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:21.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:21.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c340b3e00 con 0x7f4c3406ab70 2024-03-29T10:37:21.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 10 monclient: _finish_auth 0 2024-03-29T10:37:21.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 10 monclient: _check_auth_tickets 2024-03-29T10:37:21.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:21.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c3b1cd640 5 monclient: authenticate success, global_id 4330 2024-03-29T10:37:21.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c3b1cd640 10 monclient: _renew_subs 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c3b1cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c340b40b0 con 0x7f4c3406ab70 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c3b1cd640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c3b1cd640 10 monclient: _renew_subs 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c3b1cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c341e7990 con 0x7f4c3406ab70 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c3b1cd640 1 librados: init done 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c30001d80 con 0x7f4c3406ab70 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c2affd640 4 set_mon_vals no callback set 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4c30004aa0 con 0x7f4c3406ab70 2024-03-29T10:37:21.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.541+0000 7f4c297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.542+0000 7f4c297fa640 10 monclient: dump: 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:21.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.543+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f4c3001aa70 con 0x7f4c3406ab70 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.544+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f4c30084b40 con 0x7f4c3406ab70 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.544+0000 7f4c297fa640 10 client.4330.objecter ms_dispatch 0x7f4c34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.544+0000 7f4c297fa640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.544+0000 7f4c297fa640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.544+0000 7f4c2bfff640 10 monclient: get_auth_request con 0x7f4c2006dd50 auth_method 0 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.544+0000 7f4c297fa640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.544+0000 7f4c3b1cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.545+0000 7f4c3b1cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.545+0000 7f4c3b1cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.545+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c14005080 con 0x7f4c3406ab70 2024-03-29T10:37:21.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.545+0000 7f4c297fa640 10 client.4330.objecter ms_handle_connect 0x7f4c2006dd50 2024-03-29T10:37:21.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.553+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c30050250 con 0x7f4c3406ab70 2024-03-29T10:37:21.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.553+0000 7f4c297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:21.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.553+0000 7f4c297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:21.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.876+0000 7f4c3b1cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:21.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.877+0000 7f4c3b1cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:21.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.877+0000 7f4c3b1cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:21.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.877+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c14005330 con 0x7f4c3406ab70 2024-03-29T10:37:21.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.879+0000 7f4c297fa640 1 -- 172.21.15.117:0/2740004761 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f4c30052350 con 0x7f4c3406ab70 2024-03-29T10:37:21.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.879+0000 7f4c297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:21.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.879+0000 7f4c297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:21.864 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:37:21.864 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:21.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.883+0000 7f4c3b1cd640 10 librados: watch_flush enter 2024-03-29T10:37:21.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.883+0000 7f4c3b1cd640 10 librados: watch_flush exit 2024-03-29T10:37:21.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.883+0000 7f4c3b1cd640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T10:37:21.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.883+0000 7f4c3b1cd640 10 client.4330.objecter successfully canceled tick 2024-03-29T10:37:21.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.883+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 >> v1:172.21.15.117:6816/376768171 conn(0x7f4c2006dd50 legacy=0x7f4c20070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.883+0000 7f4c3b1cd640 10 monclient: shutdown 2024-03-29T10:37:21.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.883+0000 7f4c3b1cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:21.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.883+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 >> v1:172.21.15.136:6789/0 conn(0x7f4c3406ab70 legacy=0x7f4c341e31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.884+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 shutdown_connections 2024-03-29T10:37:21.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.884+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 >> 172.21.15.117:0/2740004761 conn(0x7f4c34067470 msgr2=0x7f4c34069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:21.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.884+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 shutdown_connections 2024-03-29T10:37:21.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.886+0000 7f4c3b1cd640 1 -- 172.21.15.117:0/2740004761 wait complete. 2024-03-29T10:37:21.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:21.886+0000 7f4c3b1cd640 1 librados: shutdown 2024-03-29T10:37:21.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:28.695 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:28.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.853+0000 7efd68f53640 10 monclient: get_monmap_and_config 2024-03-29T10:37:28.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.853+0000 7efd68f53640 10 monclient: build_initial_monmap 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.853+0000 7efd68f53640 10 monclient: monmap: 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:37:28.855211+0000 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:37:28.855211+0000 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:37:28.839 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:28.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.854+0000 7efd68f53640 1 Processor -- start 2024-03-29T10:37:28.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.855+0000 7efd68f53640 1 -- start start 2024-03-29T10:37:28.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.855+0000 7efd68f53640 10 monclient: init 2024-03-29T10:37:28.841 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.855+0000 7efd68f53640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:28.841 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.855+0000 7efd68f53640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:28.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.855+0000 7efd68f53640 10 monclient(hunting): picked mon.noname-b con 0x7efd640b2800 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:28.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.855+0000 7efd68f53640 10 monclient(hunting): picked mon.noname-c con 0x7efd640a9c00 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:28.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.855+0000 7efd68f53640 10 monclient(hunting): picked mon.noname-a con 0x7efd640b38c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:28.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.855+0000 7efd68f53640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64141eb0 con 0x7efd640b38c0 2024-03-29T10:37:28.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.856+0000 7efd68f53640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640b22d0 con 0x7efd640a9c00 2024-03-29T10:37:28.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.856+0000 7efd68f53640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64142d30 con 0x7efd640b2800 2024-03-29T10:37:28.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.856+0000 7efd68f53640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:28.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.856+0000 7efd68f53640 10 monclient(hunting): authenticate will time out at 970.119143s 2024-03-29T10:37:28.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.856+0000 7efd63fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7efd640b38c0 0x7efd6414d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:45640/0 (socket says 172.21.15.117:45640) 2024-03-29T10:37:28.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.857+0000 7efd63fff640 1 -- 172.21.15.117:0/874661806 learned_addr learned my addr 172.21.15.117:0/874661806 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:28.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.857+0000 7efd627fc640 1 -- 172.21.15.117:0/874661806 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433450233 0 0) 0x7efd64141eb0 con 0x7efd640b38c0 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.858+0000 7efd627fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.858+0000 7efd627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.858+0000 7efd627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.858+0000 7efd627fc640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.858+0000 7efd627fc640 1 -- 172.21.15.117:0/874661806 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c003520 con 0x7efd640b38c0 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.858+0000 7efd627fc640 1 -- 172.21.15.117:0/874661806 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 79032325 0 0) 0x7efd4c003520 con 0x7efd640b38c0 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.858+0000 7efd627fc640 1 -- 172.21.15.117:0/874661806 >> v1:172.21.15.117:6790/0 conn(0x7efd640b2800 legacy=0x7efd64142620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 1 -- 172.21.15.117:0/874661806 >> v1:172.21.15.136:6789/0 conn(0x7efd640a9c00 legacy=0x7efd6413bb60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 10 monclient: _finish_hunting 0 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 1 monclient: found mon.noname-a 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:28.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 1 -- 172.21.15.117:0/874661806 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd640aaf90 con 0x7efd640b38c0 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 10 monclient: _finish_auth 0 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 10 monclient: _check_auth_tickets 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 1 -- 172.21.15.117:0/874661806 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7efd5c001a10 con 0x7efd640b38c0 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.860+0000 7efd627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.861+0000 7efd627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.861+0000 7efd627fc640 10 monclient: dump: 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.848 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.861+0000 7efd627fc640 1 -- 172.21.15.117:0/874661806 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd5c001c30 con 0x7efd640b38c0 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.861+0000 7efd627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.861+0000 7efd627fc640 1 -- 172.21.15.117:0/874661806 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7efd5c001fc0 con 0x7efd640b38c0 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.861+0000 7efd627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.861+0000 7efd627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.861+0000 7efd627fc640 10 monclient: dump: 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:28.849 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.850 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.850 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:28.850 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:28.850 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:28.850 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:28.850 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:28.850 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:28.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.862+0000 7efd68f53640 5 monclient: authenticate success, global_id 4338 2024-03-29T10:37:28.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.862+0000 7efd68f53640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:28.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.862+0000 7efd68f53640 4 set_mon_vals no callback set 2024-03-29T10:37:28.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.863+0000 7efd68f53640 10 monclient: shutdown 2024-03-29T10:37:28.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.863+0000 7efd68f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.863+0000 7efd68f53640 1 -- 172.21.15.117:0/874661806 >> v1:172.21.15.117:6789/0 conn(0x7efd640b38c0 legacy=0x7efd6414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.863+0000 7efd68f53640 1 -- 172.21.15.117:0/874661806 shutdown_connections 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.863+0000 7efd68f53640 1 -- 172.21.15.117:0/874661806 >> 172.21.15.117:0/874661806 conn(0x7efd64066a00 msgr2=0x7efd64066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.864+0000 7efd68f53640 1 -- 172.21.15.117:0/874661806 shutdown_connections 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.864+0000 7efd68f53640 1 -- 172.21.15.117:0/874661806 wait complete. 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.865+0000 7efd68f53640 10 monclient: build_initial_monmap 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.865+0000 7efd68f53640 10 monclient: monmap: 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:28.853 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.866+0000 7efd68f53640 1 librados: starting msgr at 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.866+0000 7efd68f53640 1 librados: starting objecter 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.866+0000 7efd68f53640 1 Processor -- start 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.866+0000 7efd68f53640 1 -- start start 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.866+0000 7efd68f53640 1 librados: setting wanted keys 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.866+0000 7efd68f53640 1 librados: calling monclient init 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.866+0000 7efd68f53640 10 monclient: init 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:28.854 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 10 monclient(hunting): picked mon.noname-c con 0x7efd640a9c00 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 10 monclient(hunting): picked mon.noname-b con 0x7efd640b2800 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 10 monclient(hunting): picked mon.noname-a con 0x7efd640b38c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64070860 con 0x7efd640b38c0 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64071310 con 0x7efd640b2800 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6413a690 con 0x7efd640a9c00 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.867+0000 7efd68f53640 10 monclient(hunting): authenticate will time out at 970.130638s 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd637fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7efd640a9c00 0x7efd641de620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44737/0 (socket says 172.21.15.117:44737) 2024-03-29T10:37:28.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd637fe640 1 -- 172.21.15.117:0/2530307881 learned_addr learned my addr 172.21.15.117:0/2530307881 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 client.?.objecter ms_handle_connect 0x7efd640a9c00 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 client.?.objecter ms_handle_connect 0x7efd640b38c0 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381510678 0 0) 0x7efd640aaf90 con 0x7efd640b38c0 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd50004eb0 con 0x7efd640b38c0 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114918198 0 0) 0x7efd6413a690 con 0x7efd640a9c00 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:28.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.871+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd640aaf90 con 0x7efd640a9c00 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3809339009 0 0) 0x7efd50004eb0 con 0x7efd640b38c0 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd6413a690 con 0x7efd640b38c0 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7efd5c001730 con 0x7efd640b38c0 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 10 monclient(hunting): dump: 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.864 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2501186386 0 0) 0x7efd640aaf90 con 0x7efd640a9c00 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd50004eb0 con 0x7efd640a9c00 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7efd54001920 con 0x7efd640a9c00 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 10 monclient(hunting): dump: 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1523074259 0 0) 0x7efd6413a690 con 0x7efd640b38c0 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 >> v1:172.21.15.117:6790/0 conn(0x7efd640a9c00 legacy=0x7efd641de620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.872+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 >> v1:172.21.15.136:6789/0 conn(0x7efd640b2800 legacy=0x7efd64139870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.874+0000 7efd4bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.874+0000 7efd4bfff640 1 monclient: found mon.a 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd4bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd64071580 con 0x7efd640b38c0 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd4bfff640 10 monclient: _finish_auth 0 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd4bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd68f53640 5 monclient: authenticate success, global_id 4341 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd68f53640 10 monclient: _renew_subs 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd68f53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd640718c0 con 0x7efd640b38c0 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd68f53640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd68f53640 10 monclient: _renew_subs 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd68f53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd641ded30 con 0x7efd640b38c0 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.875+0000 7efd68f53640 1 librados: init done 2024-03-29T10:37:28.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.876+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd5c001a30 con 0x7efd640b38c0 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.876+0000 7efd4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.876+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7efd5c004fb0 con 0x7efd640b38c0 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.876+0000 7efd4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.876+0000 7efd68f53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.876+0000 7efd4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.876+0000 7efd4bfff640 10 monclient: dump: 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:28.868 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:28.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.876+0000 7efd61ffb640 4 set_mon_vals no callback set 2024-03-29T10:37:28.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.877+0000 7efd68f53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:28.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.877+0000 7efd68f53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:28.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.877+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd64070540 con 0x7efd640b38c0 2024-03-29T10:37:28.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.878+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7efd5c01af80 con 0x7efd640b38c0 2024-03-29T10:37:28.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.879+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7efd5c084d20 con 0x7efd640b38c0 2024-03-29T10:37:28.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.879+0000 7efd4bfff640 10 client.4341.objecter ms_dispatch 0x7efd64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:28.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.879+0000 7efd4bfff640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:28.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.879+0000 7efd4bfff640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:28.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.879+0000 7efd4bfff640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-29T10:37:28.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.879+0000 7efd637fe640 10 monclient: get_auth_request con 0x7efd5007ee50 auth_method 0 2024-03-29T10:37:28.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.879+0000 7efd4bfff640 10 client.4341.objecter ms_handle_connect 0x7efd5007ee50 2024-03-29T10:37:28.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.885+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd5c050600 con 0x7efd640b38c0 2024-03-29T10:37:28.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.885+0000 7efd4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:28.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:28.885+0000 7efd4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:29.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.209+0000 7efd68f53640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:29.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.209+0000 7efd68f53640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:29.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.209+0000 7efd68f53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:29.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.209+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd64071710 con 0x7efd640b38c0 2024-03-29T10:37:29.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.210+0000 7efd4bfff640 1 -- 172.21.15.117:0/2530307881 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7efd5c052530 con 0x7efd640b38c0 2024-03-29T10:37:29.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.210+0000 7efd4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:29.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.210+0000 7efd4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:29.196 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:37:29.196 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:29.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.213+0000 7efd68f53640 10 librados: watch_flush enter 2024-03-29T10:37:29.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.213+0000 7efd68f53640 10 librados: watch_flush exit 2024-03-29T10:37:29.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.213+0000 7efd68f53640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-29T10:37:29.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.213+0000 7efd68f53640 10 client.4341.objecter successfully canceled tick 2024-03-29T10:37:29.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.214+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 >> v1:172.21.15.117:6816/376768171 conn(0x7efd5007ee50 legacy=0x7efd50081310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:29.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.214+0000 7efd68f53640 10 monclient: shutdown 2024-03-29T10:37:29.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.214+0000 7efd68f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:29.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.214+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 >> v1:172.21.15.117:6789/0 conn(0x7efd640b38c0 legacy=0x7efd64139f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:29.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.214+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 shutdown_connections 2024-03-29T10:37:29.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.214+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 >> 172.21.15.117:0/2530307881 conn(0x7efd64066a00 msgr2=0x7efd64140470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:29.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.215+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 shutdown_connections 2024-03-29T10:37:29.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.215+0000 7efd68f53640 1 -- 172.21.15.117:0/2530307881 wait complete. 2024-03-29T10:37:29.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:29.215+0000 7efd68f53640 1 librados: shutdown 2024-03-29T10:37:29.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:36.025 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.187+0000 7f798f43e640 10 monclient: get_monmap_and_config 2024-03-29T10:37:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.187+0000 7f798f43e640 10 monclient: build_initial_monmap 2024-03-29T10:37:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.187+0000 7f798f43e640 10 monclient: monmap: 2024-03-29T10:37:36.172 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:36.172 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:36.172 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:37:36.188779+0000 2024-03-29T10:37:36.172 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:37:36.188779+0000 2024-03-29T10:37:36.173 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:36.173 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:36.173 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:36.173 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:37:36.173 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:37:36.173 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:36.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.188+0000 7f798f43e640 1 Processor -- start 2024-03-29T10:37:36.174 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.189+0000 7f798f43e640 1 -- start start 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.189+0000 7f798f43e640 10 monclient: init 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 10 monclient(hunting): picked mon.noname-c con 0x7f79880b0770 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 10 monclient(hunting): picked mon.noname-a con 0x7f798814a700 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 10 monclient(hunting): picked mon.noname-b con 0x7f79880ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79880aba70 con 0x7f798814a700 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79880aecb0 con 0x7f79880b0770 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79880af150 con 0x7f79880ac5a0 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:36.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798f43e640 10 monclient(hunting): authenticate will time out at 977.452638s 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798d1da640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f79880b0770 0x7f7988149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:38776/0 (socket says 172.21.15.117:38776) 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.190+0000 7f798d1da640 1 -- 172.21.15.117:0/4167021007 learned_addr learned my addr 172.21.15.117:0/4167021007 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.191+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045375774 0 0) 0x7f79880aecb0 con 0x7f79880b0770 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.191+0000 7f797ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.191+0000 7f797ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.191+0000 7f797ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.191+0000 7f797ffff640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.191+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7974003520 con 0x7f79880b0770 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920971295 0 0) 0x7f79880af150 con 0x7f79880ac5a0 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:37:36.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79880aecb0 con 0x7f79880ac5a0 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494976644 0 0) 0x7f7974003520 con 0x7f79880b0770 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 >> v1:172.21.15.117:6790/0 conn(0x7f79880ac5a0 legacy=0x7f7988151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 >> v1:172.21.15.117:6789/0 conn(0x7f798814a700 legacy=0x7f79880abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 1 monclient: found mon.noname-c 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79880ac9d0 con 0x7f79880b0770 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient: _finish_auth 0 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f798f43e640 5 monclient: authenticate success, global_id 4333 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f798f43e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:36.179 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f79840016b0 con 0x7f79880b0770 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient: dump: 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f798f43e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7984001890 con 0x7f79880b0770 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f797ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f798f43e640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.192+0000 7f798f43e640 4 set_mon_vals no callback set 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.193+0000 7f797ffff640 1 -- 172.21.15.117:0/4167021007 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f79840016b0 con 0x7f79880b0770 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.193+0000 7f797ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.193+0000 7f797ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.193+0000 7f797ffff640 10 monclient: dump: 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:36.181 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:36.182 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:36.182 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.194+0000 7f798f43e640 10 monclient: shutdown 2024-03-29T10:37:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.194+0000 7f798f43e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.194+0000 7f798f43e640 1 -- 172.21.15.117:0/4167021007 >> v1:172.21.15.136:6789/0 conn(0x7f79880b0770 legacy=0x7f7988149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.194+0000 7f798f43e640 1 -- 172.21.15.117:0/4167021007 shutdown_connections 2024-03-29T10:37:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.194+0000 7f798f43e640 1 -- 172.21.15.117:0/4167021007 >> 172.21.15.117:0/4167021007 conn(0x7f798806b660 msgr2=0x7f798806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.194+0000 7f798f43e640 1 -- 172.21.15.117:0/4167021007 shutdown_connections 2024-03-29T10:37:36.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.195+0000 7f798f43e640 1 -- 172.21.15.117:0/4167021007 wait complete. 2024-03-29T10:37:36.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.196+0000 7f798f43e640 10 monclient: build_initial_monmap 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.196+0000 7f798f43e640 10 monclient: monmap: 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.196+0000 7f798f43e640 1 librados: starting msgr at 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.196+0000 7f798f43e640 1 librados: starting objecter 2024-03-29T10:37:36.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.196+0000 7f798f43e640 1 Processor -- start 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.196+0000 7f798f43e640 1 -- start start 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.196+0000 7f798f43e640 1 librados: setting wanted keys 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.196+0000 7f798f43e640 1 librados: calling monclient init 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.196+0000 7f798f43e640 10 monclient: init 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 10 monclient(hunting): picked mon.noname-b con 0x7f79880ac5a0 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 10 monclient(hunting): picked mon.noname-c con 0x7f79880b0770 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 10 monclient(hunting): picked mon.noname-a con 0x7f798814a700 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79881e3f30 con 0x7f798814a700 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798811fc60 con 0x7f79880ac5a0 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79881e4b20 con 0x7f79880b0770 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798f43e640 10 monclient(hunting): authenticate will time out at 977.460455s 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798d1da640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f79880ac5a0 0x7f798811d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:39004/0 (socket says 172.21.15.117:39004) 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f798d1da640 1 -- 172.21.15.117:0/689107849 learned_addr learned my addr 172.21.15.117:0/689107849 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 client.?.objecter ms_handle_connect 0x7f79880ac5a0 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 client.?.objecter ms_handle_connect 0x7f79880b0770 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 client.?.objecter ms_handle_connect 0x7f798814a700 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255718065 0 0) 0x7f79881e4b20 con 0x7f79880b0770 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.198+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79740044b0 con 0x7f79880b0770 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644970153 0 0) 0x7f798811fc60 con 0x7f79880ac5a0 2024-03-29T10:37:36.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79881e4b20 con 0x7f79880ac5a0 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617465366 0 0) 0x7f79881e3f30 con 0x7f798814a700 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f798811fc60 con 0x7f798814a700 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 557968495 0 0) 0x7f79740044b0 con 0x7f79880b0770 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79881e3f30 con 0x7f79880b0770 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f79780017e0 con 0x7f79880b0770 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): dump: 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368291757 0 0) 0x7f79881e4b20 con 0x7f79880ac5a0 2024-03-29T10:37:36.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79740044b0 con 0x7f79880ac5a0 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7984001c20 con 0x7f79880ac5a0 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.199+0000 7f797dffb640 10 monclient(hunting): dump: 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:36.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3148153657 0 0) 0x7f79740044b0 con 0x7f79880ac5a0 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 >> v1:172.21.15.117:6790/0 conn(0x7f79880b0770 legacy=0x7f79881e4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 >> v1:172.21.15.117:6789/0 conn(0x7f798814a700 legacy=0x7f79881e0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 1 monclient: found mon.b 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79881e4cb0 con 0x7f79880ac5a0 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 10 monclient: _finish_auth 0 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f798f43e640 5 monclient: authenticate success, global_id 4336 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f798f43e640 10 monclient: _renew_subs 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f798f43e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798811b1c0 con 0x7f79880ac5a0 2024-03-29T10:37:36.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f798f43e640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f798f43e640 10 monclient: _renew_subs 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f798f43e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79881e17b0 con 0x7f79880ac5a0 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f798f43e640 1 librados: init done 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7984004a90 con 0x7f79880ac5a0 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7984004c00 con 0x7f79880ac5a0 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797dffb640 10 monclient: dump: 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.200+0000 7f797f7fe640 4 set_mon_vals no callback set 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.203+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f798401abd0 con 0x7f79880ac5a0 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.203+0000 7f798f43e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.203+0000 7f798f43e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.203+0000 7f798f43e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.203+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7968005080 con 0x7f79880ac5a0 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.204+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f7984084920 con 0x7f79880ac5a0 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.204+0000 7f797dffb640 10 client.4336.objecter ms_dispatch 0x7f7988000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.204+0000 7f797dffb640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.204+0000 7f797dffb640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.204+0000 7f797dffb640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.204+0000 7f798c9d9640 10 monclient: get_auth_request con 0x7f797406e340 auth_method 0 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.204+0000 7f797dffb640 10 client.4336.objecter ms_handle_connect 0x7f797406e340 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.211+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7984050030 con 0x7f79880ac5a0 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.211+0000 7f797dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.211+0000 7f797dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:36.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.587+0000 7f798f43e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:36.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.587+0000 7f798f43e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:36.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.587+0000 7f798f43e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:37:36.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.587+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7968005330 con 0x7f79880ac5a0 2024-03-29T10:37:36.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.588+0000 7f797dffb640 1 -- 172.21.15.117:0/689107849 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f7984052130 con 0x7f79880ac5a0 2024-03-29T10:37:36.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.588+0000 7f797dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:36.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.588+0000 7f797dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:36.573 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:37:36.573 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:36.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 10 librados: watch_flush enter 2024-03-29T10:37:36.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 10 librados: watch_flush exit 2024-03-29T10:37:36.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T10:37:36.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 10 client.4336.objecter successfully canceled tick 2024-03-29T10:37:36.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 >> v1:172.21.15.117:6816/376768171 conn(0x7f797406e340 legacy=0x7f7974070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:36.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 10 monclient: shutdown 2024-03-29T10:37:36.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:36.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 >> v1:172.21.15.136:6789/0 conn(0x7f79880ac5a0 legacy=0x7f798811d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:36.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 shutdown_connections 2024-03-29T10:37:36.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.592+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 >> 172.21.15.117:0/689107849 conn(0x7f798806b660 msgr2=0x7f79880abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:36.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.593+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 shutdown_connections 2024-03-29T10:37:36.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.593+0000 7f798f43e640 1 -- 172.21.15.117:0/689107849 wait complete. 2024-03-29T10:37:36.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:36.593+0000 7f798f43e640 1 librados: shutdown 2024-03-29T10:37:36.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:43.398 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:43.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.563+0000 7fd9d4ec8640 10 monclient: get_monmap_and_config 2024-03-29T10:37:43.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.563+0000 7fd9d4ec8640 10 monclient: build_initial_monmap 2024-03-29T10:37:43.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.563+0000 7fd9d4ec8640 10 monclient: monmap: 2024-03-29T10:37:43.548 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:43.548 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:43.548 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:37:43.564852+0000 2024-03-29T10:37:43.549 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:37:43.564852+0000 2024-03-29T10:37:43.549 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:43.549 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:43.549 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:43.549 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:37:43.549 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:37:43.549 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:43.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.564+0000 7fd9d4ec8640 1 Processor -- start 2024-03-29T10:37:43.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.565+0000 7fd9d4ec8640 1 -- start start 2024-03-29T10:37:43.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.565+0000 7fd9d4ec8640 10 monclient: init 2024-03-29T10:37:43.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:43.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:43.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d00aeda0 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:43.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d013bd60 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:43.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d0071790 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:43.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d006f550 con 0x7fd9d0071790 2024-03-29T10:37:43.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d0072330 con 0x7fd9d013bd60 2024-03-29T10:37:43.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d006fd20 con 0x7fd9d00aeda0 2024-03-29T10:37:43.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:43.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9d4ec8640 10 monclient(hunting): authenticate will time out at 984.828390s 2024-03-29T10:37:43.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9cdd9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd9d013bd60 0x7fd9d013c1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:41196/0 (socket says 172.21.15.117:41196) 2024-03-29T10:37:43.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.566+0000 7fd9cdd9b640 1 -- 172.21.15.117:0/1665457059 learned_addr learned my addr 172.21.15.117:0/1665457059 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:43.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279791188 0 0) 0x7fd9d0072330 con 0x7fd9d013bd60 2024-03-29T10:37:43.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b4003520 con 0x7fd9d013bd60 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598291644 0 0) 0x7fd9d006fd20 con 0x7fd9d00aeda0 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d0072330 con 0x7fd9d00aeda0 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071978126 0 0) 0x7fd9d006f550 con 0x7fd9d0071790 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.567+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d006fd20 con 0x7fd9d0071790 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2762914967 0 0) 0x7fd9d0072330 con 0x7fd9d00aeda0 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 >> v1:172.21.15.136:6789/0 conn(0x7fd9d013bd60 legacy=0x7fd9d013c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 >> v1:172.21.15.117:6789/0 conn(0x7fd9d0071790 legacy=0x7fd9d0071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 1 monclient: found mon.noname-b 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0145620 con 0x7fd9d00aeda0 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: _finish_auth 0 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd9c8001670 con 0x7fd9d00aeda0 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:43.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: dump: 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9d4ec8640 5 monclient: authenticate success, global_id 4370 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9d4ec8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9c8001890 con 0x7fd9d00aeda0 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:43.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9d4ec8640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9d4ec8640 4 set_mon_vals no callback set 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 1 -- 172.21.15.117:0/1665457059 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd9c8001ba0 con 0x7fd9d00aeda0 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.568+0000 7fd9cd59a640 10 monclient: dump: 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:43.558 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:43.559 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:43.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.569+0000 7fd9d4ec8640 10 monclient: shutdown 2024-03-29T10:37:43.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.569+0000 7fd9d4ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:43.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.569+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/1665457059 >> v1:172.21.15.117:6790/0 conn(0x7fd9d00aeda0 legacy=0x7fd9d006ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.569+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/1665457059 shutdown_connections 2024-03-29T10:37:43.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.569+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/1665457059 >> 172.21.15.117:0/1665457059 conn(0x7fd9d0067470 msgr2=0x7fd9d00b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:43.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.570+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/1665457059 shutdown_connections 2024-03-29T10:37:43.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.570+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/1665457059 wait complete. 2024-03-29T10:37:43.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.571+0000 7fd9d4ec8640 10 monclient: build_initial_monmap 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.571+0000 7fd9d4ec8640 10 monclient: monmap: 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.571+0000 7fd9d4ec8640 1 librados: starting msgr at 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.571+0000 7fd9d4ec8640 1 librados: starting objecter 2024-03-29T10:37:43.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.571+0000 7fd9d4ec8640 1 Processor -- start 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.571+0000 7fd9d4ec8640 1 -- start start 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.571+0000 7fd9d4ec8640 1 librados: setting wanted keys 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.571+0000 7fd9d4ec8640 1 librados: calling monclient init 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.571+0000 7fd9d4ec8640 10 monclient: init 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d0071790 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d00aeda0 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d013bd60 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d013d1c0 con 0x7fd9d00aeda0 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d013d410 con 0x7fd9d013bd60 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d013f2d0 con 0x7fd9d0071790 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9d4ec8640 10 monclient(hunting): authenticate will time out at 984.834691s 2024-03-29T10:37:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9ce59c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd9d0071790 0x7fd9d0138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:56562/0 (socket says 172.21.15.117:56562) 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9ce59c640 1 -- 172.21.15.117:0/336031411 learned_addr learned my addr 172.21.15.117:0/336031411 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9c6ffd640 10 client.?.objecter ms_handle_connect 0x7fd9d0071790 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.572+0000 7fd9c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 client.?.objecter ms_handle_connect 0x7fd9d00aeda0 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552723781 0 0) 0x7fd9d013f2d0 con 0x7fd9d0071790 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b8004eb0 con 0x7fd9d0071790 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333214988 0 0) 0x7fd9d013d1c0 con 0x7fd9d00aeda0 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d013f2d0 con 0x7fd9d00aeda0 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.573+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086755347 0 0) 0x7fd9b8004eb0 con 0x7fd9d0071790 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9d013d1c0 con 0x7fd9d0071790 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd9c8001670 con 0x7fd9d0071790 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient(hunting): dump: 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:43.570 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 249419486 0 0) 0x7fd9d013f2d0 con 0x7fd9d00aeda0 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9b8004eb0 con 0x7fd9d00aeda0 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd9bc001720 con 0x7fd9d00aeda0 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient(hunting): dump: 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:43.571 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1050753868 0 0) 0x7fd9d013d1c0 con 0x7fd9d0071790 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 >> v1:172.21.15.136:6789/0 conn(0x7fd9d013bd60 legacy=0x7fd9d01e6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 >> v1:172.21.15.117:6789/0 conn(0x7fd9d00aeda0 legacy=0x7fd9d01410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 monclient: found mon.c 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d013f5b0 con 0x7fd9d0071790 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:37:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.574+0000 7fd9c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9c80038a0 con 0x7fd9d0071790 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd9c8003bf0 con 0x7fd9d0071790 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9c6ffd640 10 monclient: dump: 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:43.573 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9d4ec8640 5 monclient: authenticate success, global_id 4373 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9d4ec8640 10 monclient: _renew_subs 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9d4ec8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d013f890 con 0x7fd9d0071790 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9d4ec8640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9d4ec8640 10 monclient: _renew_subs 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9d4ec8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d013fe90 con 0x7fd9d0071790 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9d4ec8640 1 librados: init done 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.575+0000 7fd9ccd99640 4 set_mon_vals no callback set 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.577+0000 7fd9d4ec8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.577+0000 7fd9d4ec8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.577+0000 7fd9d4ec8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:43.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.577+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d014ce60 con 0x7fd9d0071790 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.578+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fd9c8003d50 con 0x7fd9d0071790 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.579+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fd9c8084b80 con 0x7fd9d0071790 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.579+0000 7fd9c6ffd640 10 client.4373.objecter ms_dispatch 0x7fd9d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.579+0000 7fd9c6ffd640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.580+0000 7fd9c6ffd640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.580+0000 7fd9cdd9b640 10 monclient: get_auth_request con 0x7fd9b806de80 auth_method 0 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.580+0000 7fd9c6ffd640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.580+0000 7fd9c6ffd640 10 client.4373.objecter ms_handle_connect 0x7fd9b806de80 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.588+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9d014ce60 con 0x7fd9d0071790 2024-03-29T10:37:43.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.588+0000 7fd9c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:43.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.588+0000 7fd9c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:43.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.933+0000 7fd9d4ec8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:43.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.933+0000 7fd9d4ec8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:43.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.933+0000 7fd9d4ec8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:43.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.933+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9d014ce60 con 0x7fd9d0071790 2024-03-29T10:37:43.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.936+0000 7fd9c6ffd640 1 -- 172.21.15.117:0/336031411 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fd9d014ce60 con 0x7fd9d0071790 2024-03-29T10:37:43.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.936+0000 7fd9c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:43.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.936+0000 7fd9c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:43.921 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:37:43.921 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:43.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.940+0000 7fd9d4ec8640 10 librados: watch_flush enter 2024-03-29T10:37:43.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.940+0000 7fd9d4ec8640 10 librados: watch_flush exit 2024-03-29T10:37:43.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.940+0000 7fd9d4ec8640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T10:37:43.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.940+0000 7fd9d4ec8640 10 client.4373.objecter successfully canceled tick 2024-03-29T10:37:43.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.941+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 >> v1:172.21.15.117:6816/376768171 conn(0x7fd9b806de80 legacy=0x7fd9b8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.941+0000 7fd9d4ec8640 10 monclient: shutdown 2024-03-29T10:37:43.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.941+0000 7fd9d4ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:43.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.941+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 >> v1:172.21.15.117:6790/0 conn(0x7fd9d0071790 legacy=0x7fd9d0138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.941+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 shutdown_connections 2024-03-29T10:37:43.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.941+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 >> 172.21.15.117:0/336031411 conn(0x7fd9d0067470 msgr2=0x7fd9d014c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:43.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.941+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 shutdown_connections 2024-03-29T10:37:43.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.941+0000 7fd9d4ec8640 1 -- 172.21.15.117:0/336031411 wait complete. 2024-03-29T10:37:43.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:43.941+0000 7fd9d4ec8640 1 librados: shutdown 2024-03-29T10:37:43.941 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:50.748 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:50.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.917+0000 7f5e8674d640 10 monclient: get_monmap_and_config 2024-03-29T10:37:50.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.917+0000 7f5e8674d640 10 monclient: build_initial_monmap 2024-03-29T10:37:50.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.917+0000 7f5e8674d640 10 monclient: monmap: 2024-03-29T10:37:50.901 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:50.901 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:50.902 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:37:50.917956+0000 2024-03-29T10:37:50.902 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:37:50.917956+0000 2024-03-29T10:37:50.902 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:50.902 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:50.902 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:50.902 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:37:50.902 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:37:50.902 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:50.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.918+0000 7f5e8674d640 1 Processor -- start 2024-03-29T10:37:50.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.918+0000 7f5e8674d640 1 -- start start 2024-03-29T10:37:50.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.918+0000 7f5e8674d640 10 monclient: init 2024-03-29T10:37:50.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.919+0000 7f5e8674d640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:50.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.919+0000 7f5e8674d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:50.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.919+0000 7f5e8674d640 10 monclient(hunting): picked mon.noname-a con 0x7f5e78006890 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.919+0000 7f5e8674d640 10 monclient(hunting): picked mon.noname-b con 0x7f5e78006c70 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.919+0000 7f5e8674d640 10 monclient(hunting): picked mon.noname-c con 0x7f5e7802b4c0 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.919+0000 7f5e8674d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7802c830 con 0x7f5e78006890 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.919+0000 7f5e8674d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7802afa0 con 0x7f5e7802b4c0 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.919+0000 7f5e8674d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7802aba0 con 0x7f5e78006c70 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e8674d640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e8674d640 10 monclient(hunting): authenticate will time out at 992.182256s 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e84f4a640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f5e78006c70 0x7f5e780c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:50722/0 (socket says 172.21.15.117:50722) 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e84f4a640 1 -- 172.21.15.117:0/494761900 learned_addr learned my addr 172.21.15.117:0/494761900 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e77fff640 1 -- 172.21.15.117:0/494761900 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355640336 0 0) 0x7f5e7802aba0 con 0x7f5e78006c70 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e77fff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e77fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:50.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e77fff640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:37:50.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.920+0000 7f5e77fff640 1 -- 172.21.15.117:0/494761900 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e6c003520 con 0x7f5e78006c70 2024-03-29T10:37:50.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.921+0000 7f5e77fff640 1 -- 172.21.15.117:0/494761900 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1933213176 0 0) 0x7f5e6c003520 con 0x7f5e78006c70 2024-03-29T10:37:50.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.921+0000 7f5e77fff640 1 -- 172.21.15.117:0/494761900 >> v1:172.21.15.136:6789/0 conn(0x7f5e7802b4c0 legacy=0x7f5e780cf0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:37:50.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.922+0000 7f5e77fff640 1 -- 172.21.15.117:0/494761900 >> v1:172.21.15.117:6789/0 conn(0x7f5e78006890 legacy=0x7f5e780c0a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 10 monclient: _finish_hunting 0 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 1 monclient: found mon.noname-b 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 1 -- 172.21.15.117:0/494761900 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7802ad30 con 0x7f5e78006c70 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 10 monclient: _finish_auth 0 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 10 monclient: _check_auth_tickets 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 1 -- 172.21.15.117:0/494761900 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5e70001690 con 0x7f5e78006c70 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e8674d640 5 monclient: authenticate success, global_id 4376 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e8674d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:50.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.924+0000 7f5e77fff640 10 monclient: dump: 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.925+0000 7f5e8674d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.925+0000 7f5e77fff640 1 -- 172.21.15.117:0/494761900 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e700017f0 con 0x7f5e78006c70 2024-03-29T10:37:50.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.925+0000 7f5e77fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:50.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.925+0000 7f5e8674d640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:50.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.925+0000 7f5e8674d640 4 set_mon_vals no callback set 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.926+0000 7f5e8674d640 10 monclient: shutdown 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.926+0000 7f5e8674d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.926+0000 7f5e8674d640 1 -- 172.21.15.117:0/494761900 >> v1:172.21.15.117:6790/0 conn(0x7f5e78006c70 legacy=0x7f5e780c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.928+0000 7f5e8674d640 1 -- 172.21.15.117:0/494761900 shutdown_connections 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.928+0000 7f5e8674d640 1 -- 172.21.15.117:0/494761900 >> 172.21.15.117:0/494761900 conn(0x7f5e780058f0 msgr2=0x7f5e78005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.928+0000 7f5e8674d640 1 -- 172.21.15.117:0/494761900 shutdown_connections 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.928+0000 7f5e8674d640 1 -- 172.21.15.117:0/494761900 wait complete. 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.929+0000 7f5e8674d640 10 monclient: build_initial_monmap 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.929+0000 7f5e8674d640 10 monclient: monmap: 2024-03-29T10:37:50.913 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:50.914 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:50.914 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:37:50.914 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:37:50.914 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:50.914 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:50.914 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:50.914 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:37:50.914 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:37:50.914 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:50.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 1 librados: starting msgr at 2024-03-29T10:37:50.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 1 librados: starting objecter 2024-03-29T10:37:50.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 1 Processor -- start 2024-03-29T10:37:50.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 1 -- start start 2024-03-29T10:37:50.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 1 librados: setting wanted keys 2024-03-29T10:37:50.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 1 librados: calling monclient init 2024-03-29T10:37:50.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 10 monclient: init 2024-03-29T10:37:50.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:50.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:37:50.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.930+0000 7f5e8674d640 10 monclient(hunting): picked mon.noname-b con 0x7f5e78006890 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:50.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.931+0000 7f5e8674d640 10 monclient(hunting): picked mon.noname-c con 0x7f5e7802b4c0 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:50.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.931+0000 7f5e8674d640 10 monclient(hunting): picked mon.noname-a con 0x7f5e780bb430 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:50.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.931+0000 7f5e8674d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e780bb9e0 con 0x7f5e780bb430 2024-03-29T10:37:50.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.931+0000 7f5e8674d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e780ba320 con 0x7f5e78006890 2024-03-29T10:37:50.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.931+0000 7f5e8674d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e780bce60 con 0x7f5e7802b4c0 2024-03-29T10:37:50.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.931+0000 7f5e8674d640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:50.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.931+0000 7f5e8674d640 10 monclient(hunting): authenticate will time out at 992.193136s 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.934+0000 7f5e85f4c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f5e780bb430 0x7f5e780bf0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:49943/0 (socket says 172.21.15.117:49943) 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.934+0000 7f5e85f4c640 1 -- 172.21.15.117:0/747573786 learned_addr learned my addr 172.21.15.117:0/747573786 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 10 client.?.objecter ms_handle_connect 0x7f5e780bb430 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193789982 0 0) 0x7f5e780bb9e0 con 0x7f5e780bb430 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e64004eb0 con 0x7f5e780bb430 2024-03-29T10:37:50.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 10 client.?.objecter ms_handle_connect 0x7f5e7802b4c0 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.935+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259595111 0 0) 0x7f5e780bce60 con 0x7f5e7802b4c0 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e780bb9e0 con 0x7f5e7802b4c0 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1467248258 0 0) 0x7f5e64004eb0 con 0x7f5e780bb430 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e780bce60 con 0x7f5e780bb430 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5e8004fdd0 con 0x7f5e780bb430 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 10 monclient(hunting): dump: 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:50.923 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.936+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1361085078 0 0) 0x7f5e780bce60 con 0x7f5e780bb430 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.937+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 >> v1:172.21.15.117:6790/0 conn(0x7f5e7802b4c0 legacy=0x7f5e780be9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.937+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 >> v1:172.21.15.136:6789/0 conn(0x7f5e78006890 legacy=0x7f5e780bad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.937+0000 7f5e75ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:37:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.937+0000 7f5e75ffb640 1 monclient: found mon.a 2024-03-29T10:37:50.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.937+0000 7f5e75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:50.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.937+0000 7f5e75ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:50.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.937+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e78178000 con 0x7f5e780bb430 2024-03-29T10:37:50.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.937+0000 7f5e75ffb640 10 monclient: _finish_auth 0 2024-03-29T10:37:50.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e75ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:37:50.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:50.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e8674d640 5 monclient: authenticate success, global_id 4356 2024-03-29T10:37:50.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e8674d640 10 monclient: _renew_subs 2024-03-29T10:37:50.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e8674d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:50.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e78174fd0 con 0x7f5e780bb430 2024-03-29T10:37:50.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e8674d640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:50.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e8674d640 10 monclient: _renew_subs 2024-03-29T10:37:50.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e8674d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e78175460 con 0x7f5e780bb430 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e8674d640 1 librados: init done 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e8004f400 con 0x7f5e780bb430 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.938+0000 7f5e777fe640 4 set_mon_vals no callback set 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.940+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5e80075710 con 0x7f5e780bb430 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.940+0000 7f5e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.940+0000 7f5e75ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.940+0000 7f5e75ffb640 10 monclient: dump: 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:50.928 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:50.929 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:50.929 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:50.929 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:50.929 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:50.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.940+0000 7f5e8674d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:50.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.940+0000 7f5e8674d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:50.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.940+0000 7f5e8674d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:50.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.940+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e60005080 con 0x7f5e780bb430 2024-03-29T10:37:50.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.940+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f5e80067bf0 con 0x7f5e780bb430 2024-03-29T10:37:50.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.943+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f5e800ff540 con 0x7f5e780bb430 2024-03-29T10:37:50.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.943+0000 7f5e75ffb640 10 client.4356.objecter ms_dispatch 0x7f5e80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:50.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.943+0000 7f5e75ffb640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:50.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.943+0000 7f5e75ffb640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:50.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.943+0000 7f5e75ffb640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T10:37:50.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.945+0000 7f5e8574b640 10 monclient: get_auth_request con 0x7f5e6406d910 auth_method 0 2024-03-29T10:37:50.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.946+0000 7f5e75ffb640 10 client.4356.objecter ms_handle_connect 0x7f5e6406d910 2024-03-29T10:37:50.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.952+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e800cae20 con 0x7f5e780bb430 2024-03-29T10:37:50.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.952+0000 7f5e75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:50.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:50.952+0000 7f5e75ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:51.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.278+0000 7f5e8674d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:51.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.278+0000 7f5e8674d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:51.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.278+0000 7f5e8674d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:37:51.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.278+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e60005330 con 0x7f5e780bb430 2024-03-29T10:37:51.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.279+0000 7f5e75ffb640 1 -- 172.21.15.117:0/747573786 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f5e800ccd50 con 0x7f5e780bb430 2024-03-29T10:37:51.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.279+0000 7f5e75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:51.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.279+0000 7f5e75ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:51.264 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:37:51.264 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:51.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.282+0000 7f5e8674d640 10 librados: watch_flush enter 2024-03-29T10:37:51.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.283+0000 7f5e8674d640 10 librados: watch_flush exit 2024-03-29T10:37:51.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.283+0000 7f5e8674d640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T10:37:51.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.283+0000 7f5e8674d640 10 client.4356.objecter successfully canceled tick 2024-03-29T10:37:51.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.283+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 >> v1:172.21.15.117:6816/376768171 conn(0x7f5e6406d910 legacy=0x7f5e6406fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:51.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.283+0000 7f5e8674d640 10 monclient: shutdown 2024-03-29T10:37:51.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.283+0000 7f5e8674d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:51.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.283+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 >> v1:172.21.15.117:6789/0 conn(0x7f5e780bb430 legacy=0x7f5e780bf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:51.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.283+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 shutdown_connections 2024-03-29T10:37:51.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.283+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 >> 172.21.15.117:0/747573786 conn(0x7f5e780058f0 msgr2=0x7f5e78038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:51.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.285+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 shutdown_connections 2024-03-29T10:37:51.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.285+0000 7f5e8674d640 1 -- 172.21.15.117:0/747573786 wait complete. 2024-03-29T10:37:51.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:51.285+0000 7f5e8674d640 1 librados: shutdown 2024-03-29T10:37:51.289 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:58.096 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.260+0000 7ffb7c0ba640 10 monclient: get_monmap_and_config 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.260+0000 7ffb7c0ba640 10 monclient: build_initial_monmap 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.260+0000 7ffb7c0ba640 10 monclient: monmap: 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:37:58.261727+0000 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:37:58.261727+0000 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:58.246 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:37:58.247 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:37:58.247 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:58.247 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.261+0000 7ffb7c0ba640 1 Processor -- start 2024-03-29T10:37:58.247 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.262+0000 7ffb7c0ba640 1 -- start start 2024-03-29T10:37:58.247 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.262+0000 7ffb7c0ba640 10 monclient: init 2024-03-29T10:37:58.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.262+0000 7ffb7c0ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:58.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.262+0000 7ffb7c0ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:58.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.262+0000 7ffb7c0ba640 10 monclient(hunting): picked mon.noname-b con 0x7ffb7406d9d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.262+0000 7ffb7c0ba640 10 monclient(hunting): picked mon.noname-c con 0x7ffb7413ef30 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.263+0000 7ffb7c0ba640 10 monclient(hunting): picked mon.noname-a con 0x7ffb740b0860 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.263+0000 7ffb7c0ba640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb740aed10 con 0x7ffb740b0860 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.263+0000 7ffb7c0ba640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb740ab1b0 con 0x7ffb7413ef30 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.263+0000 7ffb7c0ba640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb740ab490 con 0x7ffb7406d9d0 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.263+0000 7ffb7c0ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.263+0000 7ffb7c0ba640 10 monclient(hunting): authenticate will time out at 999.525242s 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.264+0000 7ffb79e56640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ffb7406d9d0 0x7ffb7413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:45948/0 (socket says 172.21.15.117:45948) 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.264+0000 7ffb79e56640 1 -- 172.21.15.117:0/2042028584 learned_addr learned my addr 172.21.15.117:0/2042028584 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.267+0000 7ffb78e54640 1 -- 172.21.15.117:0/2042028584 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415015106 0 0) 0x7ffb740ab490 con 0x7ffb7406d9d0 2024-03-29T10:37:58.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.267+0000 7ffb78e54640 10 monclient(hunting): _negotiate 2024-03-29T10:37:58.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.267+0000 7ffb78e54640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:58.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.267+0000 7ffb78e54640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:58.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.267+0000 7ffb78e54640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:37:58.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.267+0000 7ffb78e54640 1 -- 172.21.15.117:0/2042028584 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb60003520 con 0x7ffb7406d9d0 2024-03-29T10:37:58.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.268+0000 7ffb78e54640 1 -- 172.21.15.117:0/2042028584 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2418846313 0 0) 0x7ffb60003520 con 0x7ffb7406d9d0 2024-03-29T10:37:58.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.268+0000 7ffb78e54640 1 -- 172.21.15.117:0/2042028584 >> v1:172.21.15.136:6789/0 conn(0x7ffb7413ef30 legacy=0x7ffb740af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:37:58.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.268+0000 7ffb78e54640 1 -- 172.21.15.117:0/2042028584 >> v1:172.21.15.117:6789/0 conn(0x7ffb740b0860 legacy=0x7ffb7414ab80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:37:58.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 10 monclient: _finish_hunting 0 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 1 monclient: found mon.noname-b 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 1 -- 172.21.15.117:0/2042028584 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb740ab770 con 0x7ffb7406d9d0 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 10 monclient: _finish_auth 0 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 10 monclient: _check_auth_tickets 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb7c0ba640 5 monclient: authenticate success, global_id 4382 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb7c0ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 1 -- 172.21.15.117:0/2042028584 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ffb6c001a10 con 0x7ffb7406d9d0 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb78e54640 10 monclient: dump: 2024-03-29T10:37:58.257 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.271+0000 7ffb7c0ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.272+0000 7ffb78e54640 1 -- 172.21.15.117:0/2042028584 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb6c001be0 con 0x7ffb7406d9d0 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.272+0000 7ffb78e54640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.272+0000 7ffb78e54640 1 -- 172.21.15.117:0/2042028584 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ffb6c002030 con 0x7ffb7406d9d0 2024-03-29T10:37:58.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.272+0000 7ffb7c0ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.272+0000 7ffb7c0ba640 4 set_mon_vals no callback set 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.272+0000 7ffb78e54640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.272+0000 7ffb78e54640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.272+0000 7ffb78e54640 10 monclient: dump: 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:58.259 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:58.260 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:58.260 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:58.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.273+0000 7ffb7c0ba640 10 monclient: shutdown 2024-03-29T10:37:58.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.273+0000 7ffb7c0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:58.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.273+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2042028584 >> v1:172.21.15.117:6790/0 conn(0x7ffb7406d9d0 legacy=0x7ffb7413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.274+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2042028584 shutdown_connections 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.274+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2042028584 >> 172.21.15.117:0/2042028584 conn(0x7ffb7406b660 msgr2=0x7ffb7406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.274+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2042028584 shutdown_connections 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.274+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2042028584 wait complete. 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.275+0000 7ffb7c0ba640 10 monclient: build_initial_monmap 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.275+0000 7ffb7c0ba640 10 monclient: monmap: 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:37:58.263 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.275+0000 7ffb7c0ba640 1 librados: starting msgr at 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.275+0000 7ffb7c0ba640 1 librados: starting objecter 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 1 Processor -- start 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 1 -- start start 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 1 librados: setting wanted keys 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 1 librados: calling monclient init 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 10 monclient: init 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 10 monclient(hunting): picked mon.noname-b con 0x7ffb7406d9d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 10 monclient(hunting): picked mon.noname-a con 0x7ffb740b0860 addr v1:172.21.15.117:6789/0 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 10 monclient(hunting): picked mon.noname-c con 0x7ffb7413ef30 addr v1:172.21.15.117:6790/0 2024-03-29T10:37:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb741db530 con 0x7ffb740b0860 2024-03-29T10:37:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb740b1a50 con 0x7ffb7406d9d0 2024-03-29T10:37:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7413a180 con 0x7ffb7413ef30 2024-03-29T10:37:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.276+0000 7ffb7c0ba640 10 monclient(hunting): authenticate will time out at 999.539049s 2024-03-29T10:37:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.278+0000 7ffb7a657640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ffb7413ef30 0x7ffb741dee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:49580/0 (socket says 172.21.15.117:49580) 2024-03-29T10:37:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.278+0000 7ffb7a657640 1 -- 172.21.15.117:0/2052183521 learned_addr learned my addr 172.21.15.117:0/2052183521 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:37:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.279+0000 7ffb6a7fc640 10 client.?.objecter ms_handle_connect 0x7ffb7413ef30 2024-03-29T10:37:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.279+0000 7ffb6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871714968 0 0) 0x7ffb7413a180 con 0x7ffb7413ef30 2024-03-29T10:37:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb58004eb0 con 0x7ffb7413ef30 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035718557 0 0) 0x7ffb58004eb0 con 0x7ffb7413ef30 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb58005300 con 0x7ffb7413ef30 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ffb70003fc0 con 0x7ffb7413ef30 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 10 monclient(hunting): dump: 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:58.270 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4028655721 0 0) 0x7ffb58005300 con 0x7ffb7413ef30 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.280+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 >> v1:172.21.15.136:6789/0 conn(0x7ffb7406d9d0 legacy=0x7ffb7413acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 >> v1:172.21.15.117:6789/0 conn(0x7ffb740b0860 legacy=0x7ffb741dbb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb6a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb6a7fc640 1 monclient: found mon.c 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb6a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb741dc220 con 0x7ffb7413ef30 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb6a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb6a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb7c0ba640 5 monclient: authenticate success, global_id 4385 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb7c0ba640 10 monclient: _renew_subs 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb7c0ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb74133560 con 0x7ffb7413ef30 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb7c0ba640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb7c0ba640 10 monclient: _renew_subs 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb7c0ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb741df700 con 0x7ffb7413ef30 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.281+0000 7ffb7c0ba640 1 librados: init done 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.282+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb70000ed0 con 0x7ffb7413ef30 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.282+0000 7ffb6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.282+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ffb70003fc0 con 0x7ffb7413ef30 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.282+0000 7ffb6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.282+0000 7ffb6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.282+0000 7ffb6a7fc640 10 monclient: dump: 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:37:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.282+0000 7ffb6bfff640 4 set_mon_vals no callback set 2024-03-29T10:37:58.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.283+0000 7ffb7c0ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:58.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.283+0000 7ffb7c0ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:58.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.283+0000 7ffb7c0ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:58.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.283+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c005080 con 0x7ffb7413ef30 2024-03-29T10:37:58.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.287+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7ffb70001140 con 0x7ffb7413ef30 2024-03-29T10:37:58.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.288+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7ffb70084730 con 0x7ffb7413ef30 2024-03-29T10:37:58.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.288+0000 7ffb6a7fc640 10 client.4385.objecter ms_dispatch 0x7ffb74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:58.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.288+0000 7ffb6a7fc640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:58.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.288+0000 7ffb6a7fc640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:58.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.288+0000 7ffb6a7fc640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T10:37:58.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.289+0000 7ffb79e56640 10 monclient: get_auth_request con 0x7ffb5806e020 auth_method 0 2024-03-29T10:37:58.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.289+0000 7ffb6a7fc640 10 client.4385.objecter ms_handle_connect 0x7ffb5806e020 2024-03-29T10:37:58.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.291+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb70050010 con 0x7ffb7413ef30 2024-03-29T10:37:58.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.291+0000 7ffb6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:58.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.291+0000 7ffb6a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:58.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.611+0000 7ffb7c0ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:58.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.611+0000 7ffb7c0ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:58.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.611+0000 7ffb7c0ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:37:58.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.611+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb5c005330 con 0x7ffb7413ef30 2024-03-29T10:37:58.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.613+0000 7ffb6a7fc640 1 -- 172.21.15.117:0/2052183521 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7ffb70051f40 con 0x7ffb7413ef30 2024-03-29T10:37:58.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.613+0000 7ffb6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:58.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.613+0000 7ffb6a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:58.598 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:37:58.598 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:58.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.617+0000 7ffb7c0ba640 10 librados: watch_flush enter 2024-03-29T10:37:58.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.617+0000 7ffb7c0ba640 10 librados: watch_flush exit 2024-03-29T10:37:58.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.617+0000 7ffb7c0ba640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T10:37:58.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.617+0000 7ffb7c0ba640 10 client.4385.objecter successfully canceled tick 2024-03-29T10:37:58.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.617+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 >> v1:172.21.15.117:6816/376768171 conn(0x7ffb5806e020 legacy=0x7ffb580704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:58.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.617+0000 7ffb7c0ba640 10 monclient: shutdown 2024-03-29T10:37:58.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.617+0000 7ffb7c0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:58.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.617+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 >> v1:172.21.15.117:6790/0 conn(0x7ffb7413ef30 legacy=0x7ffb741dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:58.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.618+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 shutdown_connections 2024-03-29T10:37:58.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.618+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 >> 172.21.15.117:0/2052183521 conn(0x7ffb7406b660 msgr2=0x7ffb7413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:58.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.618+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 shutdown_connections 2024-03-29T10:37:58.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.618+0000 7ffb7c0ba640 1 -- 172.21.15.117:0/2052183521 wait complete. 2024-03-29T10:37:58.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:37:58.619+0000 7ffb7c0ba640 1 librados: shutdown 2024-03-29T10:37:58.620 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:05.428 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.594+0000 7f0b2b348640 10 monclient: get_monmap_and_config 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.594+0000 7f0b2b348640 10 monclient: build_initial_monmap 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.594+0000 7f0b2b348640 10 monclient: monmap: 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:38:05.595617+0000 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:38:05.595617+0000 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:38:05.579 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:38:05.580 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:05.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.595+0000 7f0b2b348640 1 Processor -- start 2024-03-29T10:38:05.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.596+0000 7f0b2b348640 1 -- start start 2024-03-29T10:38:05.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.596+0000 7f0b2b348640 10 monclient: init 2024-03-29T10:38:05.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.596+0000 7f0b2b348640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:05.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.597+0000 7f0b2b348640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:05.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.600+0000 7f0b2b348640 10 monclient(hunting): picked mon.noname-c con 0x7f0b241457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:05.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.600+0000 7f0b2b348640 10 monclient(hunting): picked mon.noname-a con 0x7f0b2413f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:05.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.600+0000 7f0b2b348640 10 monclient(hunting): picked mon.noname-b con 0x7f0b240a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.600+0000 7f0b2b348640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b240aaef0 con 0x7f0b2413f650 2024-03-29T10:38:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.602+0000 7f0b2b348640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b24145bd0 con 0x7f0b241457f0 2024-03-29T10:38:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.602+0000 7f0b2b348640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b24145eb0 con 0x7f0b240a9d70 2024-03-29T10:38:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.602+0000 7f0b2b348640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.602+0000 7f0b2b348640 10 monclient(hunting): authenticate will time out at 1006.864712s 2024-03-29T10:38:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.604+0000 7f0b288e3640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0b2413f650 0x7f0b240b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:41623/0 (socket says 172.21.15.117:41623) 2024-03-29T10:38:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.605+0000 7f0b288e3640 1 -- 172.21.15.117:0/4161615740 learned_addr learned my addr 172.21.15.117:0/4161615740 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.605+0000 7f0b1bfff640 1 -- 172.21.15.117:0/4161615740 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853840755 0 0) 0x7f0b240aaef0 con 0x7f0b2413f650 2024-03-29T10:38:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.605+0000 7f0b1bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.605+0000 7f0b1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:05.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.605+0000 7f0b1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:05.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.605+0000 7f0b1bfff640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:38:05.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.605+0000 7f0b1bfff640 1 -- 172.21.15.117:0/4161615740 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b10003520 con 0x7f0b2413f650 2024-03-29T10:38:05.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.605+0000 7f0b1bfff640 1 -- 172.21.15.117:0/4161615740 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1030184935 0 0) 0x7f0b10003520 con 0x7f0b2413f650 2024-03-29T10:38:05.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.605+0000 7f0b1bfff640 1 -- 172.21.15.117:0/4161615740 >> v1:172.21.15.117:6790/0 conn(0x7f0b240a9d70 legacy=0x7f0b2414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 1 -- 172.21.15.117:0/4161615740 >> v1:172.21.15.136:6789/0 conn(0x7f0b241457f0 legacy=0x7f0b2413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 1 monclient: found mon.noname-a 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 1 -- 172.21.15.117:0/4161615740 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24146340 con 0x7f0b2413f650 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 10 monclient: _finish_auth 0 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 1 -- 172.21.15.117:0/4161615740 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0b14001690 con 0x7f0b2413f650 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b1bfff640 10 monclient: dump: 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b2b348640 5 monclient: authenticate success, global_id 4359 2024-03-29T10:38:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.606+0000 7f0b2b348640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.607+0000 7f0b1bfff640 1 -- 172.21.15.117:0/4161615740 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b140018f0 con 0x7f0b2413f650 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.607+0000 7f0b1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.607+0000 7f0b2b348640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.607+0000 7f0b2b348640 4 set_mon_vals no callback set 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.608+0000 7f0b2b348640 10 monclient: shutdown 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.608+0000 7f0b1bfff640 1 -- 172.21.15.117:0/4161615740 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0b14001d00 con 0x7f0b2413f650 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.608+0000 7f0b1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.608+0000 7f0b1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.608+0000 7f0b1bfff640 10 monclient: dump: 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:05.595 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:05.596 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:05.596 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:05.596 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:05.596 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:05.596 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:05.596 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.609+0000 7f0b2b348640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.609+0000 7f0b2b348640 1 -- 172.21.15.117:0/4161615740 >> v1:172.21.15.117:6789/0 conn(0x7f0b2413f650 legacy=0x7f0b240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.610+0000 7f0b2b348640 1 -- 172.21.15.117:0/4161615740 shutdown_connections 2024-03-29T10:38:05.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.611+0000 7f0b2b348640 1 -- 172.21.15.117:0/4161615740 >> 172.21.15.117:0/4161615740 conn(0x7f0b24066a00 msgr2=0x7f0b24066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:05.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.614+0000 7f0b2b348640 1 -- 172.21.15.117:0/4161615740 shutdown_connections 2024-03-29T10:38:05.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.615+0000 7f0b2b348640 1 -- 172.21.15.117:0/4161615740 wait complete. 2024-03-29T10:38:05.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.615+0000 7f0b2b348640 10 monclient: build_initial_monmap 2024-03-29T10:38:05.601 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.615+0000 7f0b2b348640 10 monclient: monmap: 2024-03-29T10:38:05.601 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:05.601 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:05.601 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:38:05.601 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:38:05.601 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:05.601 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:05.601 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:05.602 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:38:05.602 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:38:05.602 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:05.602 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.616+0000 7f0b2b348640 1 librados: starting msgr at 2024-03-29T10:38:05.602 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.616+0000 7f0b2b348640 1 librados: starting objecter 2024-03-29T10:38:05.602 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.616+0000 7f0b2b348640 1 Processor -- start 2024-03-29T10:38:05.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.618+0000 7f0b2b348640 1 -- start start 2024-03-29T10:38:05.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.618+0000 7f0b2b348640 1 librados: setting wanted keys 2024-03-29T10:38:05.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.619+0000 7f0b2b348640 1 librados: calling monclient init 2024-03-29T10:38:05.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.619+0000 7f0b2b348640 10 monclient: init 2024-03-29T10:38:05.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.620+0000 7f0b2b348640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:05.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.620+0000 7f0b2b348640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.620+0000 7f0b2b348640 10 monclient(hunting): picked mon.noname-a con 0x7f0b240a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.621+0000 7f0b2b348640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2413f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.621+0000 7f0b2b348640 10 monclient(hunting): picked mon.noname-c con 0x7f0b241457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.621+0000 7f0b2b348640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2406f760 con 0x7f0b240a9d70 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.621+0000 7f0b2b348640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2406e810 con 0x7f0b2413f650 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.621+0000 7f0b2b348640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b241424b0 con 0x7f0b241457f0 2024-03-29T10:38:05.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.621+0000 7f0b2b348640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:05.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.621+0000 7f0b2b348640 10 monclient(hunting): authenticate will time out at 1006.883611s 2024-03-29T10:38:05.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.626+0000 7f0b288e3640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0b2413f650 0x7f0b24140180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:53763/0 (socket says 172.21.15.117:53763) 2024-03-29T10:38:05.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b288e3640 1 -- 172.21.15.117:0/63950029 learned_addr learned my addr 172.21.15.117:0/63950029 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:05.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 client.?.objecter ms_handle_connect 0x7f0b2413f650 2024-03-29T10:38:05.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:05.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251360891 0 0) 0x7f0b2406e810 con 0x7f0b2413f650 2024-03-29T10:38:05.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:05.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:05.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:05.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:38:05.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b08004eb0 con 0x7f0b2413f650 2024-03-29T10:38:05.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 client.?.objecter ms_handle_connect 0x7f0b241457f0 2024-03-29T10:38:05.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:05.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 client.?.objecter ms_handle_connect 0x7f0b240a9d70 2024-03-29T10:38:05.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.627+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247522127 0 0) 0x7f0b241424b0 con 0x7f0b241457f0 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2406e810 con 0x7f0b241457f0 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917707254 0 0) 0x7f0b2406f760 con 0x7f0b240a9d70 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.628+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b08005360 con 0x7f0b240a9d70 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3249798321 0 0) 0x7f0b08005360 con 0x7f0b240a9d70 2024-03-29T10:38:05.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b2406f760 con 0x7f0b240a9d70 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0b1c001b40 con 0x7f0b240a9d70 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 10 monclient(hunting): dump: 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:05.617 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3505110471 0 0) 0x7f0b2406f760 con 0x7f0b240a9d70 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 >> v1:172.21.15.117:6790/0 conn(0x7f0b241457f0 legacy=0x7f0b24140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 >> v1:172.21.15.136:6789/0 conn(0x7f0b2413f650 legacy=0x7f0b24140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 1 monclient: found mon.a 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24142640 con 0x7f0b240a9d70 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 10 monclient: _finish_auth 0 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b2b348640 5 monclient: authenticate success, global_id 4362 2024-03-29T10:38:05.618 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.629+0000 7f0b2b348640 10 monclient: _renew_subs 2024-03-29T10:38:05.619 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.630+0000 7f0b2b348640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:05.619 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.630+0000 7f0b2b348640 1 -- 172.21.15.117:0/63950029 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b241ef9e0 con 0x7f0b240a9d70 2024-03-29T10:38:05.619 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.630+0000 7f0b2b348640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:05.619 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.630+0000 7f0b2b348640 10 monclient: _renew_subs 2024-03-29T10:38:05.619 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.630+0000 7f0b2b348640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:05.619 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.630+0000 7f0b2b348640 1 -- 172.21.15.117:0/63950029 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b241efd50 con 0x7f0b240a9d70 2024-03-29T10:38:05.619 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.630+0000 7f0b2b348640 1 librados: init done 2024-03-29T10:38:05.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.632+0000 7f0b0f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:05.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.632+0000 7f0b0f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:05.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.632+0000 7f0b0f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:05.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.632+0000 7f0b0f7fe640 1 -- 172.21.15.117:0/63950029 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b24075f10 con 0x7f0b240a9d70 2024-03-29T10:38:05.623 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.637+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b1c001ff0 con 0x7f0b240a9d70 2024-03-29T10:38:05.623 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.637+0000 7f0b19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:05.623 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.637+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0b1c003b10 con 0x7f0b240a9d70 2024-03-29T10:38:05.623 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.637+0000 7f0b19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:05.623 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.637+0000 7f0b19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.637+0000 7f0b19ffb640 10 monclient: dump: 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.637+0000 7f0b1b7fe640 4 set_mon_vals no callback set 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.638+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f0b1c003c70 con 0x7f0b240a9d70 2024-03-29T10:38:05.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.639+0000 7f0b288e3640 10 monclient: get_auth_request con 0x7f0b0806de80 auth_method 0 2024-03-29T10:38:05.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.639+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f0b1c053770 con 0x7f0b240a9d70 2024-03-29T10:38:05.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.639+0000 7f0b19ffb640 10 client.4362.objecter ms_dispatch 0x7f0b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:05.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.639+0000 7f0b19ffb640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:05.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.639+0000 7f0b19ffb640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:05.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.639+0000 7f0b19ffb640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T10:38:05.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.639+0000 7f0b19ffb640 10 client.4362.objecter ms_handle_connect 0x7f0b0806de80 2024-03-29T10:38:05.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.643+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b1c04fa10 con 0x7f0b240a9d70 2024-03-29T10:38:05.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.643+0000 7f0b19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:05.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.643+0000 7f0b19ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:05.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.962+0000 7f0b0f7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:05.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.962+0000 7f0b0f7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:05.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.962+0000 7f0b0f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:05.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.962+0000 7f0b0f7fe640 1 -- 172.21.15.117:0/63950029 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b24072080 con 0x7f0b240a9d70 2024-03-29T10:38:05.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.966+0000 7f0b19ffb640 1 -- 172.21.15.117:0/63950029 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f0b1c04fbb0 con 0x7f0b240a9d70 2024-03-29T10:38:05.951 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:38:05.951 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:05.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.966+0000 7f0b19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:05.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.966+0000 7f0b19ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.969+0000 7f0b0f7fe640 10 librados: watch_flush enter 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.969+0000 7f0b0f7fe640 10 librados: watch_flush exit 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.969+0000 7f0b0f7fe640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.969+0000 7f0b0f7fe640 10 client.4362.objecter successfully canceled tick 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.970+0000 7f0b0f7fe640 1 -- 172.21.15.117:0/63950029 >> v1:172.21.15.117:6816/376768171 conn(0x7f0b0806de80 legacy=0x7f0b08070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.970+0000 7f0b0f7fe640 10 monclient: shutdown 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.970+0000 7f0b0f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.970+0000 7f0b0f7fe640 1 -- 172.21.15.117:0/63950029 >> v1:172.21.15.117:6789/0 conn(0x7f0b240a9d70 legacy=0x7f0b2406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.970+0000 7f0b0f7fe640 1 -- 172.21.15.117:0/63950029 shutdown_connections 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.970+0000 7f0b0f7fe640 1 -- 172.21.15.117:0/63950029 >> 172.21.15.117:0/63950029 conn(0x7f0b24066a00 msgr2=0x7f0b24139020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.970+0000 7f0b0f7fe640 1 -- 172.21.15.117:0/63950029 shutdown_connections 2024-03-29T10:38:05.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.970+0000 7f0b0f7fe640 1 -- 172.21.15.117:0/63950029 wait complete. 2024-03-29T10:38:05.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:05.972+0000 7f0b0f7fe640 1 librados: shutdown 2024-03-29T10:38:05.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:12.782 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.951+0000 7f8010c9a640 10 monclient: get_monmap_and_config 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.951+0000 7f8010c9a640 10 monclient: build_initial_monmap 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.951+0000 7f8010c9a640 10 monclient: monmap: 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:38:12.951932+0000 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:38:12.951932+0000 2024-03-29T10:38:12.936 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:12.936 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:12.936 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:12.936 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:38:12.936 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:38:12.936 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:12.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.952+0000 7f8010c9a640 1 Processor -- start 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.952+0000 7f8010c9a640 1 -- start start 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.952+0000 7f8010c9a640 10 monclient: init 2024-03-29T10:38:12.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.953+0000 7f8010c9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:12.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.953+0000 7f8010c9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:12.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.953+0000 7f8010c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f800c0b0770 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:12.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.953+0000 7f8010c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f800c14a700 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:12.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.955+0000 7f8010c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f800c0ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:12.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.955+0000 7f8010c9a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0aba70 con 0x7f800c0b0770 2024-03-29T10:38:12.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.955+0000 7f8010c9a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0aecb0 con 0x7f800c14a700 2024-03-29T10:38:12.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.955+0000 7f8010c9a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c0af150 con 0x7f800c0ac5a0 2024-03-29T10:38:12.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.955+0000 7f8010c9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:12.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.955+0000 7f8010c9a640 10 monclient(hunting): authenticate will time out at 1014.217981s 2024-03-29T10:38:12.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.957+0000 7f800a59c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f800c0b0770 0x7f800c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:38037/0 (socket says 172.21.15.117:38037) 2024-03-29T10:38:12.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.957+0000 7f800a59c640 1 -- 172.21.15.117:0/4219252517 learned_addr learned my addr 172.21.15.117:0/4219252517 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.957+0000 7f8009d9b640 1 -- 172.21.15.117:0/4219252517 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566455647 0 0) 0x7f800c0aba70 con 0x7f800c0b0770 2024-03-29T10:38:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.958+0000 7f8009d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:38:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.958+0000 7f8009d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.958+0000 7f8009d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.958+0000 7f8009d9b640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:38:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.958+0000 7f8009d9b640 1 -- 172.21.15.117:0/4219252517 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff8003520 con 0x7f800c0b0770 2024-03-29T10:38:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.958+0000 7f8009d9b640 1 -- 172.21.15.117:0/4219252517 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463240403 0 0) 0x7f7ff8003520 con 0x7f800c0b0770 2024-03-29T10:38:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.958+0000 7f8009d9b640 1 -- 172.21.15.117:0/4219252517 >> v1:172.21.15.117:6790/0 conn(0x7f800c0ac5a0 legacy=0x7f800c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.959+0000 7f8009d9b640 1 -- 172.21.15.117:0/4219252517 >> v1:172.21.15.136:6789/0 conn(0x7f800c14a700 legacy=0x7f800c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.959+0000 7f8009d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.959+0000 7f8009d9b640 1 monclient: found mon.noname-a 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.959+0000 7f8009d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.959+0000 7f8009d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 1 -- 172.21.15.117:0/4219252517 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c0ac9d0 con 0x7f800c0b0770 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 10 monclient: _finish_auth 0 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 1 -- 172.21.15.117:0/4219252517 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7ffc001690 con 0x7f800c0b0770 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8010c9a640 5 monclient: authenticate success, global_id 4365 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8010c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:12.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 10 monclient: dump: 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8010c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 1 -- 172.21.15.117:0/4219252517 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ffc001920 con 0x7f800c0b0770 2024-03-29T10:38:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8009d9b640 1 -- 172.21.15.117:0/4219252517 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7ffc001cb0 con 0x7f800c0b0770 2024-03-29T10:38:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8010c9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.960+0000 7f8010c9a640 4 set_mon_vals no callback set 2024-03-29T10:38:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.961+0000 7f8009d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.961+0000 7f8009d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.961+0000 7f8009d9b640 10 monclient: dump: 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.961+0000 7f8010c9a640 10 monclient: shutdown 2024-03-29T10:38:12.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.961+0000 7f8010c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:12.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.961+0000 7f8010c9a640 1 -- 172.21.15.117:0/4219252517 >> v1:172.21.15.117:6789/0 conn(0x7f800c0b0770 legacy=0x7f800c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:12.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.962+0000 7f8010c9a640 1 -- 172.21.15.117:0/4219252517 shutdown_connections 2024-03-29T10:38:12.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.962+0000 7f8010c9a640 1 -- 172.21.15.117:0/4219252517 >> 172.21.15.117:0/4219252517 conn(0x7f800c06b660 msgr2=0x7f800c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:12.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.962+0000 7f8010c9a640 1 -- 172.21.15.117:0/4219252517 shutdown_connections 2024-03-29T10:38:12.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.962+0000 7f8010c9a640 1 -- 172.21.15.117:0/4219252517 wait complete. 2024-03-29T10:38:12.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.963+0000 7f8010c9a640 10 monclient: build_initial_monmap 2024-03-29T10:38:12.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.963+0000 7f8010c9a640 10 monclient: monmap: 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.963+0000 7f8010c9a640 1 librados: starting msgr at 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.963+0000 7f8010c9a640 1 librados: starting objecter 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.963+0000 7f8010c9a640 1 Processor -- start 2024-03-29T10:38:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.963+0000 7f8010c9a640 1 -- start start 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.963+0000 7f8010c9a640 1 librados: setting wanted keys 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.963+0000 7f8010c9a640 1 librados: calling monclient init 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.963+0000 7f8010c9a640 10 monclient: init 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.964+0000 7f8010c9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.964+0000 7f8010c9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.964+0000 7f8010c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f800c0ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.964+0000 7f8010c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f800c0b0770 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.966+0000 7f8010c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f800c14a700 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.966+0000 7f8010c9a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c1efb80 con 0x7f800c0b0770 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.966+0000 7f8010c9a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c1f07b0 con 0x7f800c14a700 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.966+0000 7f8010c9a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c07e420 con 0x7f800c0ac5a0 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.966+0000 7f8010c9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.966+0000 7f8010c9a640 10 monclient(hunting): authenticate will time out at 1014.228303s 2024-03-29T10:38:12.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.969+0000 7f800a59c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f800c0ac5a0 0x7f800c07cf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41473/0 (socket says 172.21.15.117:41473) 2024-03-29T10:38:12.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.969+0000 7f800a59c640 1 -- 172.21.15.117:0/420713694 learned_addr learned my addr 172.21.15.117:0/420713694 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:12.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.972+0000 7f8002ffd640 10 client.?.objecter ms_handle_connect 0x7f800c0ac5a0 2024-03-29T10:38:12.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.972+0000 7f8002ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.973+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773774832 0 0) 0x7f800c07e420 con 0x7f800c0ac5a0 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.973+0000 7f8002ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.973+0000 7f8002ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.973+0000 7f8002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.973+0000 7f8002ffd640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.973+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fec004eb0 con 0x7f800c0ac5a0 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.973+0000 7f8002ffd640 10 client.?.objecter ms_handle_connect 0x7f800c0b0770 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.973+0000 7f8002ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148246609 0 0) 0x7f800c1efb80 con 0x7f800c0b0770 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fec0051d0 con 0x7f800c0b0770 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2038148157 0 0) 0x7f7fec0051d0 con 0x7f800c0b0770 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f800c1efb80 con 0x7f800c0b0770 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7ff4001850 con 0x7f800c0b0770 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.974+0000 7f8002ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient(hunting): dump: 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:12.961 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 client.?.objecter ms_handle_connect 0x7f800c14a700 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 819302569 0 0) 0x7f800c1efb80 con 0x7f800c0b0770 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 >> v1:172.21.15.117:6790/0 conn(0x7f800c0ac5a0 legacy=0x7f800c07cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 >> v1:172.21.15.136:6789/0 conn(0x7f800c14a700 legacy=0x7f800c1e95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 1 monclient: found mon.a 2024-03-29T10:38:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c07e5b0 con 0x7f800c0b0770 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient: _finish_auth 0 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8010c9a640 5 monclient: authenticate success, global_id 4368 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8010c9a640 10 monclient: _renew_subs 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8010c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c07aff0 con 0x7f800c0b0770 2024-03-29T10:38:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8010c9a640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8010c9a640 10 monclient: _renew_subs 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8010c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ff4001d40 con 0x7f800c0b0770 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c07b340 con 0x7f800c0b0770 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8010c9a640 1 librados: init done 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7ff4003790 con 0x7f800c0b0770 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f800959a640 4 set_mon_vals no callback set 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.975+0000 7f8002ffd640 10 monclient: dump: 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:12.967 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.977+0000 7f8010c9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.977+0000 7f8010c9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.977+0000 7f8010c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.977+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800c07dd30 con 0x7f800c0b0770 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.979+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f7ff4003970 con 0x7f800c0b0770 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.980+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f7ff4083670 con 0x7f800c0b0770 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.980+0000 7f8002ffd640 10 client.4368.objecter ms_dispatch 0x7f800c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.980+0000 7f8002ffd640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.980+0000 7f8002ffd640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.980+0000 7f8002ffd640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T10:38:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.981+0000 7f800a59c640 10 monclient: get_auth_request con 0x7f7fec06e160 auth_method 0 2024-03-29T10:38:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.981+0000 7f8002ffd640 10 client.4368.objecter ms_handle_connect 0x7f7fec06e160 2024-03-29T10:38:12.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.988+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ff404ed80 con 0x7f800c0b0770 2024-03-29T10:38:12.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.988+0000 7f8002ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:12.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:12.988+0000 7f8002ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:13.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.319+0000 7f8010c9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:13.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.319+0000 7f8010c9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:13.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.319+0000 7f8010c9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:13.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.319+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f800c075f10 con 0x7f800c0b0770 2024-03-29T10:38:13.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.320+0000 7f8002ffd640 1 -- 172.21.15.117:0/420713694 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f7ff404ed80 con 0x7f800c0b0770 2024-03-29T10:38:13.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.320+0000 7f8002ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:13.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.320+0000 7f8002ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:13.310 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:38:13.310 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:13.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.326+0000 7f8010c9a640 10 librados: watch_flush enter 2024-03-29T10:38:13.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.326+0000 7f8010c9a640 10 librados: watch_flush exit 2024-03-29T10:38:13.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.326+0000 7f8010c9a640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T10:38:13.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.326+0000 7f8010c9a640 10 client.4368.objecter successfully canceled tick 2024-03-29T10:38:13.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.327+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 >> v1:172.21.15.117:6816/376768171 conn(0x7f7fec06e160 legacy=0x7f7fec070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:13.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.327+0000 7f8010c9a640 10 monclient: shutdown 2024-03-29T10:38:13.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.327+0000 7f8010c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:13.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.327+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 >> v1:172.21.15.117:6789/0 conn(0x7f800c0b0770 legacy=0x7f800c1f00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:13.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.332+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 shutdown_connections 2024-03-29T10:38:13.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.332+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 >> 172.21.15.117:0/420713694 conn(0x7f800c06b660 msgr2=0x7f800c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:13.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.332+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 shutdown_connections 2024-03-29T10:38:13.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.332+0000 7f8010c9a640 1 -- 172.21.15.117:0/420713694 wait complete. 2024-03-29T10:38:13.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:13.332+0000 7f8010c9a640 1 librados: shutdown 2024-03-29T10:38:13.334 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:20.141 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.312+0000 7f2244eac640 10 monclient: get_monmap_and_config 2024-03-29T10:38:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.312+0000 7f2244eac640 10 monclient: build_initial_monmap 2024-03-29T10:38:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.312+0000 7f2244eac640 10 monclient: monmap: 2024-03-29T10:38:20.298 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:20.298 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:20.298 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:38:20.314732+0000 2024-03-29T10:38:20.298 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:38:20.314732+0000 2024-03-29T10:38:20.298 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:20.298 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:20.299 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:20.299 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:38:20.299 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:38:20.299 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.314+0000 7f2244eac640 1 Processor -- start 2024-03-29T10:38:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.314+0000 7f2244eac640 1 -- start start 2024-03-29T10:38:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.314+0000 7f2244eac640 10 monclient: init 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 10 monclient(hunting): picked mon.noname-a con 0x7f22400b4820 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 10 monclient(hunting): picked mon.noname-c con 0x7f22400a9ce0 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 10 monclient(hunting): picked mon.noname-b con 0x7f22400b3cd0 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400b4180 con 0x7f22400b4820 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400aaf60 con 0x7f22400a9ce0 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224013c200 con 0x7f22400b3cd0 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.315+0000 7f2244eac640 10 monclient(hunting): authenticate will time out at 1021.578545s 2024-03-29T10:38:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.318+0000 7f223ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f22400b3cd0 0x7f224014d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:33419/0 (socket says 172.21.15.117:33419) 2024-03-29T10:38:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.319+0000 7f223ffff640 1 -- 172.21.15.117:0/1210396808 learned_addr learned my addr 172.21.15.117:0/1210396808 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:20.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380599364 0 0) 0x7f224013c200 con 0x7f22400b3cd0 2024-03-29T10:38:20.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:20.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:20.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:20.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c003520 con 0x7f22400b3cd0 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970546821 0 0) 0x7f22300013c0 con 0x7f22400b4820 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.323+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f224013c200 con 0x7f22400b4820 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.324+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315011399 0 0) 0x7f22400aaf60 con 0x7f22400a9ce0 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.324+0000 7f223e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.324+0000 7f223e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.324+0000 7f223e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.324+0000 7f223e7fc640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.324+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22300013c0 con 0x7f22400a9ce0 2024-03-29T10:38:20.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.324+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2676214121 0 0) 0x7f222c003520 con 0x7f22400b3cd0 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 >> v1:172.21.15.136:6789/0 conn(0x7f22400a9ce0 legacy=0x7f2240145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 >> v1:172.21.15.117:6789/0 conn(0x7f22400b4820 legacy=0x7f224013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2240145e80 con 0x7f22400b3cd0 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2238001690 con 0x7f22400b3cd0 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:20.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: dump: 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f2244eac640 5 monclient: authenticate success, global_id 4394 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f2244eac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22380017f0 con 0x7f22400b3cd0 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:20.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f2244eac640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f2244eac640 4 set_mon_vals no callback set 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 1 -- 172.21.15.117:0/1210396808 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2238001c00 con 0x7f22400b3cd0 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.325+0000 7f223e7fc640 10 monclient: dump: 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:20.315 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:20.316 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:20.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.326+0000 7f2244eac640 10 monclient: shutdown 2024-03-29T10:38:20.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.326+0000 7f2244eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:20.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.326+0000 7f2244eac640 1 -- 172.21.15.117:0/1210396808 >> v1:172.21.15.117:6790/0 conn(0x7f22400b3cd0 legacy=0x7f224014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:20.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.326+0000 7f2244eac640 1 -- 172.21.15.117:0/1210396808 shutdown_connections 2024-03-29T10:38:20.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.326+0000 7f2244eac640 1 -- 172.21.15.117:0/1210396808 >> 172.21.15.117:0/1210396808 conn(0x7f224006d070 msgr2=0x7f2240066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:20.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.329+0000 7f2244eac640 1 -- 172.21.15.117:0/1210396808 shutdown_connections 2024-03-29T10:38:20.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.329+0000 7f2244eac640 1 -- 172.21.15.117:0/1210396808 wait complete. 2024-03-29T10:38:20.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.330+0000 7f2244eac640 10 monclient: build_initial_monmap 2024-03-29T10:38:20.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.330+0000 7f2244eac640 10 monclient: monmap: 2024-03-29T10:38:20.317 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:20.317 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:20.317 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.330+0000 7f2244eac640 1 librados: starting msgr at 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.330+0000 7f2244eac640 1 librados: starting objecter 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.330+0000 7f2244eac640 1 Processor -- start 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.331+0000 7f2244eac640 1 -- start start 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.331+0000 7f2244eac640 1 librados: setting wanted keys 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.331+0000 7f2244eac640 1 librados: calling monclient init 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.331+0000 7f2244eac640 10 monclient: init 2024-03-29T10:38:20.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.332+0000 7f2244eac640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:20.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.332+0000 7f2244eac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:20.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.332+0000 7f2244eac640 10 monclient(hunting): picked mon.noname-b con 0x7f22400a9ce0 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:20.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.332+0000 7f2244eac640 10 monclient(hunting): picked mon.noname-a con 0x7f22400b3cd0 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.332+0000 7f2244eac640 10 monclient(hunting): picked mon.noname-c con 0x7f22400b4820 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.339+0000 7f2244eac640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400ad310 con 0x7f22400b3cd0 2024-03-29T10:38:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.339+0000 7f223f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f22400a9ce0 0x7f22401e6df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:43101/0 (socket says 172.21.15.117:43101) 2024-03-29T10:38:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.339+0000 7f223f7fe640 1 -- 172.21.15.117:0/3921012679 learned_addr learned my addr 172.21.15.117:0/3921012679 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.339+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400aea50 con 0x7f22400a9ce0 2024-03-29T10:38:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.339+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224013f610 con 0x7f22400b4820 2024-03-29T10:38:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.339+0000 7f2244eac640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:20.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.339+0000 7f2244eac640 10 monclient(hunting): authenticate will time out at 1021.602539s 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.340+0000 7f2237fff640 10 client.?.objecter ms_handle_connect 0x7f22400b4820 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.340+0000 7f2237fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.340+0000 7f2237fff640 10 client.?.objecter ms_handle_connect 0x7f22400b3cd0 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.340+0000 7f2237fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.340+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428301864 0 0) 0x7f22400ad310 con 0x7f22400b3cd0 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.340+0000 7f2237fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.340+0000 7f2237fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.340+0000 7f2237fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.340+0000 7f2237fff640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:38:20.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.341+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2220004eb0 con 0x7f22400b3cd0 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.341+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161751184 0 0) 0x7f224013f610 con 0x7f22400b4820 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.341+0000 7f2237fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.341+0000 7f2237fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.341+0000 7f2237fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22400ad310 con 0x7f22400b4820 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 10 client.?.objecter ms_handle_connect 0x7f22400a9ce0 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4125531332 0 0) 0x7f22400ad310 con 0x7f22400b4820 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f224013f610 con 0x7f22400b4820 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2238001b00 con 0x7f22400b4820 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:20.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.342+0000 7f2237fff640 10 monclient(hunting): dump: 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372255318 0 0) 0x7f22400aea50 con 0x7f22400a9ce0 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:20.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22400ad310 con 0x7f22400a9ce0 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3632545743 0 0) 0x7f224013f610 con 0x7f22400b4820 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 >> v1:172.21.15.136:6789/0 conn(0x7f22400a9ce0 legacy=0x7f22401e6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 >> v1:172.21.15.117:6789/0 conn(0x7f22400b3cd0 legacy=0x7f224013e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 1 monclient: found mon.c 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22400ad8c0 con 0x7f22400b4820 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient: _finish_auth 0 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:20.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2244eac640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2244eac640 10 monclient: _renew_subs 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2244eac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22400adb10 con 0x7f22400b4820 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2244eac640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2244eac640 10 monclient: _renew_subs 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2244eac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22400acf00 con 0x7f22400b4820 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2244eac640 1 librados: init done 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2238004a90 con 0x7f22400b4820 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2238001250 con 0x7f22400b4820 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:20.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.343+0000 7f2237fff640 10 monclient: dump: 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:20.333 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.344+0000 7f2244eac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.344+0000 7f2244eac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.345+0000 7f2244eac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.345+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224004f730 con 0x7f22400b4820 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.346+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f223801ac10 con 0x7f22400b4820 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.346+0000 7f223dffb640 4 set_mon_vals no callback set 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.346+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f2238084a20 con 0x7f22400b4820 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.346+0000 7f2237fff640 10 client.4397.objecter ms_dispatch 0x7f2240000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.346+0000 7f2237fff640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.346+0000 7f2237fff640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.346+0000 7f2237fff640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.347+0000 7f223f7fe640 10 monclient: get_auth_request con 0x7f222006dd90 auth_method 0 2024-03-29T10:38:20.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.347+0000 7f2237fff640 10 client.4397.objecter ms_handle_connect 0x7f222006dd90 2024-03-29T10:38:20.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.355+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2238050130 con 0x7f22400b4820 2024-03-29T10:38:20.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.355+0000 7f2237fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:20.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.355+0000 7f2237fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:20.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.676+0000 7f2244eac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:20.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.676+0000 7f2244eac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:20.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.676+0000 7f2244eac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:20.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.676+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2240074d80 con 0x7f22400b4820 2024-03-29T10:38:20.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.680+0000 7f2237fff640 1 -- 172.21.15.117:0/3921012679 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f2238052230 con 0x7f22400b4820 2024-03-29T10:38:20.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.680+0000 7f2237fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:20.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.680+0000 7f2237fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:20.665 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:38:20.666 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:20.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.683+0000 7f2244eac640 10 librados: watch_flush enter 2024-03-29T10:38:20.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.684+0000 7f2244eac640 10 librados: watch_flush exit 2024-03-29T10:38:20.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.684+0000 7f2244eac640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T10:38:20.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.684+0000 7f2244eac640 10 client.4397.objecter successfully canceled tick 2024-03-29T10:38:20.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.684+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 >> v1:172.21.15.117:6816/376768171 conn(0x7f222006dd90 legacy=0x7f2220070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:20.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.684+0000 7f2244eac640 10 monclient: shutdown 2024-03-29T10:38:20.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.684+0000 7f2244eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:20.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.684+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 >> v1:172.21.15.117:6790/0 conn(0x7f22400b4820 legacy=0x7f224013ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:20.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.684+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 shutdown_connections 2024-03-29T10:38:20.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.684+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 >> 172.21.15.117:0/3921012679 conn(0x7f224006d070 msgr2=0x7f22400b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:20.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.686+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 shutdown_connections 2024-03-29T10:38:20.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.686+0000 7f2244eac640 1 -- 172.21.15.117:0/3921012679 wait complete. 2024-03-29T10:38:20.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:20.686+0000 7f2244eac640 1 librados: shutdown 2024-03-29T10:38:20.688 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:27.495 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:27.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.658+0000 7f9a7ea74640 10 monclient: get_monmap_and_config 2024-03-29T10:38:27.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.658+0000 7f9a7ea74640 10 monclient: build_initial_monmap 2024-03-29T10:38:27.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.658+0000 7f9a7ea74640 10 monclient: monmap: 2024-03-29T10:38:27.642 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:27.643 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:27.643 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:38:27.659068+0000 2024-03-29T10:38:27.643 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:38:27.659068+0000 2024-03-29T10:38:27.643 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:27.643 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:27.643 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:27.643 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:38:27.643 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:38:27.643 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.659+0000 7f9a7ea74640 1 Processor -- start 2024-03-29T10:38:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.659+0000 7f9a7ea74640 1 -- start start 2024-03-29T10:38:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.659+0000 7f9a7ea74640 10 monclient: init 2024-03-29T10:38:27.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:27.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:27.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 10 monclient(hunting): picked mon.noname-c con 0x7f9a781422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:27.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 10 monclient(hunting): picked mon.noname-a con 0x7f9a78142690 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:27.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 10 monclient(hunting): picked mon.noname-b con 0x7f9a780b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:27.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a78142c80 con 0x7f9a78142690 2024-03-29T10:38:27.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a780aae40 con 0x7f9a781422b0 2024-03-29T10:38:27.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7813fbd0 con 0x7f9a780b17d0 2024-03-29T10:38:27.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:27.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.660+0000 7f9a7ea74640 10 monclient(hunting): authenticate will time out at 1028.922756s 2024-03-29T10:38:27.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.663+0000 7f9a77fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f9a78142690 0x7f9a780b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:33647/0 (socket says 172.21.15.117:33647) 2024-03-29T10:38:27.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.663+0000 7f9a77fff640 1 -- 172.21.15.117:0/1322465341 learned_addr learned my addr 172.21.15.117:0/1322465341 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:27.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.664+0000 7f9a777fe640 1 -- 172.21.15.117:0/1322465341 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241486337 0 0) 0x7f9a78142c80 con 0x7f9a78142690 2024-03-29T10:38:27.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.664+0000 7f9a777fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:27.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.664+0000 7f9a777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:27.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.664+0000 7f9a777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:27.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.664+0000 7f9a777fe640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:38:27.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.664+0000 7f9a777fe640 1 -- 172.21.15.117:0/1322465341 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64003520 con 0x7f9a78142690 2024-03-29T10:38:27.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.665+0000 7f9a777fe640 1 -- 172.21.15.117:0/1322465341 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 875493674 0 0) 0x7f9a64003520 con 0x7f9a78142690 2024-03-29T10:38:27.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.665+0000 7f9a777fe640 1 -- 172.21.15.117:0/1322465341 >> v1:172.21.15.117:6790/0 conn(0x7f9a780b17d0 legacy=0x7f9a780b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:27.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.665+0000 7f9a777fe640 1 -- 172.21.15.117:0/1322465341 >> v1:172.21.15.136:6789/0 conn(0x7f9a781422b0 legacy=0x7f9a7813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:27.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.665+0000 7f9a777fe640 10 monclient: _finish_hunting 0 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.665+0000 7f9a777fe640 1 monclient: found mon.noname-a 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.665+0000 7f9a777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.665+0000 7f9a777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.665+0000 7f9a777fe640 1 -- 172.21.15.117:0/1322465341 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a78151940 con 0x7f9a78142690 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 10 monclient: _finish_auth 0 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 10 monclient: _check_auth_tickets 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 1 -- 172.21.15.117:0/1322465341 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f9a60001690 con 0x7f9a78142690 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 10 monclient: dump: 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:27.651 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a7ea74640 5 monclient: authenticate success, global_id 4377 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a7ea74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 1 -- 172.21.15.117:0/1322465341 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a600017f0 con 0x7f9a78142690 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 1 -- 172.21.15.117:0/1322465341 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f9a60001c40 con 0x7f9a78142690 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a777fe640 10 monclient: dump: 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a7ea74640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:27.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.666+0000 7f9a7ea74640 4 set_mon_vals no callback set 2024-03-29T10:38:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.667+0000 7f9a7ea74640 10 monclient: shutdown 2024-03-29T10:38:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.667+0000 7f9a7ea74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.667+0000 7f9a7ea74640 1 -- 172.21.15.117:0/1322465341 >> v1:172.21.15.117:6789/0 conn(0x7f9a78142690 legacy=0x7f9a780b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.667+0000 7f9a7ea74640 1 -- 172.21.15.117:0/1322465341 shutdown_connections 2024-03-29T10:38:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.667+0000 7f9a7ea74640 1 -- 172.21.15.117:0/1322465341 >> 172.21.15.117:0/1322465341 conn(0x7f9a78066a00 msgr2=0x7f9a78066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.669+0000 7f9a7ea74640 1 -- 172.21.15.117:0/1322465341 shutdown_connections 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.671+0000 7f9a7ea74640 1 -- 172.21.15.117:0/1322465341 wait complete. 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.672+0000 7f9a7ea74640 10 monclient: build_initial_monmap 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.672+0000 7f9a7ea74640 10 monclient: monmap: 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:27.657 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:27.658 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:38:27.658 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:38:27.658 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.672+0000 7f9a7ea74640 1 librados: starting msgr at 2024-03-29T10:38:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.672+0000 7f9a7ea74640 1 librados: starting objecter 2024-03-29T10:38:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.672+0000 7f9a7ea74640 1 Processor -- start 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.673+0000 7f9a7ea74640 1 -- start start 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.673+0000 7f9a7ea74640 1 librados: setting wanted keys 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.673+0000 7f9a7ea74640 1 librados: calling monclient init 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.673+0000 7f9a7ea74640 10 monclient: init 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 10 monclient(hunting): picked mon.noname-a con 0x7f9a780b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 10 monclient(hunting): picked mon.noname-b con 0x7f9a781422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 10 monclient(hunting): picked mon.noname-c con 0x7f9a78142690 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a78132280 con 0x7f9a780b17d0 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7806f5d0 con 0x7f9a781422b0 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7806f8b0 con 0x7f9a78142690 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.674+0000 7f9a7ea74640 10 monclient(hunting): authenticate will time out at 1028.936598s 2024-03-29T10:38:27.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.678+0000 7f9a77fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9a781422b0 0x7f9a7812c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:60619/0 (socket says 172.21.15.117:60619) 2024-03-29T10:38:27.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.678+0000 7f9a77fff640 1 -- 172.21.15.117:0/1121021402 learned_addr learned my addr 172.21.15.117:0/1121021402 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:27.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.679+0000 7f9a757fa640 10 client.?.objecter ms_handle_connect 0x7f9a781422b0 2024-03-29T10:38:27.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.679+0000 7f9a757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:27.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.679+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451635672 0 0) 0x7f9a7806f5d0 con 0x7f9a781422b0 2024-03-29T10:38:27.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.679+0000 7f9a757fa640 10 monclient(hunting): _negotiate 2024-03-29T10:38:27.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.679+0000 7f9a757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:27.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.679+0000 7f9a757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:27.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.679+0000 7f9a757fa640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:38:27.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.679+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a58004eb0 con 0x7f9a781422b0 2024-03-29T10:38:27.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.681+0000 7f9a757fa640 10 client.?.objecter ms_handle_connect 0x7f9a780b17d0 2024-03-29T10:38:27.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.682+0000 7f9a757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:27.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.682+0000 7f9a757fa640 10 client.?.objecter ms_handle_connect 0x7f9a78142690 2024-03-29T10:38:27.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.682+0000 7f9a757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:27.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.682+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2402160655 0 0) 0x7f9a58004eb0 con 0x7f9a781422b0 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.682+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a58005300 con 0x7f9a781422b0 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.682+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f9a60001c00 con 0x7f9a781422b0 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.682+0000 7f9a757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.682+0000 7f9a757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 10 monclient(hunting): dump: 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:27.667 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:27.668 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:27.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:27.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1129402517 0 0) 0x7f9a58005300 con 0x7f9a781422b0 2024-03-29T10:38:27.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 >> v1:172.21.15.117:6790/0 conn(0x7f9a78142690 legacy=0x7f9a7812c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:27.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 >> v1:172.21.15.117:6789/0 conn(0x7f9a780b17d0 legacy=0x7f9a7806e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:27.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 10 monclient: _finish_hunting 0 2024-03-29T10:38:27.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 1 monclient: found mon.b 2024-03-29T10:38:27.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:27.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a78132a80 con 0x7f9a781422b0 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 10 monclient: _finish_auth 0 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 10 monclient: _check_auth_tickets 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a7ea74640 5 monclient: authenticate success, global_id 4360 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a7ea74640 10 monclient: _renew_subs 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a7ea74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a7ea74640 1 -- 172.21.15.117:0/1121021402 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a781efa50 con 0x7f9a781422b0 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a7ea74640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a7ea74640 10 monclient: _renew_subs 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a7ea74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a7ea74640 1 -- 172.21.15.117:0/1121021402 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a781efdc0 con 0x7f9a781422b0 2024-03-29T10:38:27.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.683+0000 7f9a7ea74640 1 librados: init done 2024-03-29T10:38:27.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.685+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a60004a90 con 0x7f9a781422b0 2024-03-29T10:38:27.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.685+0000 7f9a757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.685+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f9a60001c00 con 0x7f9a781422b0 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.685+0000 7f9a757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.685+0000 7f9a757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.685+0000 7f9a757fa640 10 monclient: dump: 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:27.672 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.685+0000 7f9a76ffd640 4 set_mon_vals no callback set 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.685+0000 7f9a6effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.686+0000 7f9a6effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.686+0000 7f9a6effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.686+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f9a6001aaf0 con 0x7f9a781422b0 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.686+0000 7f9a6effd640 1 -- 172.21.15.117:0/1121021402 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a78075f10 con 0x7f9a781422b0 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.687+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f9a60054530 con 0x7f9a781422b0 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.687+0000 7f9a757fa640 10 client.4360.objecter ms_dispatch 0x7f9a78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.687+0000 7f9a757fa640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.687+0000 7f9a757fa640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:27.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.687+0000 7f9a757fa640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T10:38:27.674 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.689+0000 7f9a7c810640 10 monclient: get_auth_request con 0x7f9a5806e320 auth_method 0 2024-03-29T10:38:27.674 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.690+0000 7f9a757fa640 10 client.4360.objecter ms_handle_connect 0x7f9a5806e320 2024-03-29T10:38:27.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.695+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a600507d0 con 0x7f9a781422b0 2024-03-29T10:38:27.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.695+0000 7f9a757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:27.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:27.695+0000 7f9a757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:28.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.028+0000 7f9a6effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:28.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.028+0000 7f9a6effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:28.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.028+0000 7f9a6effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:28.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.028+0000 7f9a6effd640 1 -- 172.21.15.117:0/1121021402 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a78075f10 con 0x7f9a781422b0 2024-03-29T10:38:28.017 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:38:28.017 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:28.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.032+0000 7f9a757fa640 1 -- 172.21.15.117:0/1121021402 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f9a60050970 con 0x7f9a781422b0 2024-03-29T10:38:28.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.032+0000 7f9a757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:28.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.032+0000 7f9a757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:28.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.036+0000 7f9a6effd640 10 librados: watch_flush enter 2024-03-29T10:38:28.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.036+0000 7f9a6effd640 10 librados: watch_flush exit 2024-03-29T10:38:28.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.036+0000 7f9a6effd640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T10:38:28.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.036+0000 7f9a6effd640 10 client.4360.objecter successfully canceled tick 2024-03-29T10:38:28.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.036+0000 7f9a6effd640 1 -- 172.21.15.117:0/1121021402 >> v1:172.21.15.117:6816/376768171 conn(0x7f9a5806e320 legacy=0x7f9a580707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:28.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.036+0000 7f9a6effd640 10 monclient: shutdown 2024-03-29T10:38:28.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.036+0000 7f9a6effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:28.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.036+0000 7f9a6effd640 1 -- 172.21.15.117:0/1121021402 >> v1:172.21.15.136:6789/0 conn(0x7f9a781422b0 legacy=0x7f9a7812c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:28.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.037+0000 7f9a6effd640 1 -- 172.21.15.117:0/1121021402 shutdown_connections 2024-03-29T10:38:28.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.037+0000 7f9a6effd640 1 -- 172.21.15.117:0/1121021402 >> 172.21.15.117:0/1121021402 conn(0x7f9a78066a00 msgr2=0x7f9a780b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:28.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.037+0000 7f9a6effd640 1 -- 172.21.15.117:0/1121021402 shutdown_connections 2024-03-29T10:38:28.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.037+0000 7f9a6effd640 1 -- 172.21.15.117:0/1121021402 wait complete. 2024-03-29T10:38:28.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:28.038+0000 7f9a6effd640 1 librados: shutdown 2024-03-29T10:38:28.043 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:34.850 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:35.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.021+0000 7f1b16e99640 10 monclient: get_monmap_and_config 2024-03-29T10:38:35.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.021+0000 7f1b16e99640 10 monclient: build_initial_monmap 2024-03-29T10:38:35.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.022+0000 7f1b16e99640 10 monclient: monmap: 2024-03-29T10:38:35.006 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:35.006 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:35.006 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:38:35.022901+0000 2024-03-29T10:38:35.006 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:38:35.022901+0000 2024-03-29T10:38:35.006 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:35.006 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:35.007 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:35.007 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:38:35.007 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:38:35.007 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:35.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.026+0000 7f1b16e99640 1 Processor -- start 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.026+0000 7f1b16e99640 1 -- start start 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.026+0000 7f1b16e99640 10 monclient: init 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 10 monclient(hunting): picked mon.noname-c con 0x7f1b080389e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 10 monclient(hunting): picked mon.noname-b con 0x7f1b08038dc0 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 10 monclient(hunting): picked mon.noname-a con 0x7f1b08030660 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b080c54c0 con 0x7f1b08030660 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b08036430 con 0x7f1b080389e0 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b08039840 con 0x7f1b08038dc0 2024-03-29T10:38:35.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:35.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.027+0000 7f1b16e99640 10 monclient(hunting): authenticate will time out at 1036.289657s 2024-03-29T10:38:35.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.030+0000 7f1b16698640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1b08030660 0x7f1b080d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:44109/0 (socket says 172.21.15.117:44109) 2024-03-29T10:38:35.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.030+0000 7f1b16698640 1 -- 172.21.15.117:0/3732050542 learned_addr learned my addr 172.21.15.117:0/3732050542 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:35.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.031+0000 7f1b14e95640 1 -- 172.21.15.117:0/3732050542 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968385018 0 0) 0x7f1b080c54c0 con 0x7f1b08030660 2024-03-29T10:38:35.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.031+0000 7f1b14e95640 10 monclient(hunting): _negotiate 2024-03-29T10:38:35.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.031+0000 7f1b14e95640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:35.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.031+0000 7f1b14e95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:35.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.031+0000 7f1b14e95640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:38:35.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.031+0000 7f1b14e95640 1 -- 172.21.15.117:0/3732050542 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1afc003520 con 0x7f1b08030660 2024-03-29T10:38:35.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.032+0000 7f1b14e95640 1 -- 172.21.15.117:0/3732050542 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1809483799 0 0) 0x7f1afc003520 con 0x7f1b08030660 2024-03-29T10:38:35.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.032+0000 7f1b14e95640 1 -- 172.21.15.117:0/3732050542 >> v1:172.21.15.117:6790/0 conn(0x7f1b08038dc0 legacy=0x7f1b080cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:35.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.034+0000 7f1b14e95640 1 -- 172.21.15.117:0/3732050542 >> v1:172.21.15.136:6789/0 conn(0x7f1b080389e0 legacy=0x7f1b08033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:35.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 10 monclient: _finish_hunting 0 2024-03-29T10:38:35.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 1 monclient: found mon.noname-a 2024-03-29T10:38:35.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:35.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:38:35.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 1 -- 172.21.15.117:0/3732050542 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b08035690 con 0x7f1b08030660 2024-03-29T10:38:35.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 10 monclient: _finish_auth 0 2024-03-29T10:38:35.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 10 monclient: _check_auth_tickets 2024-03-29T10:38:35.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 1 -- 172.21.15.117:0/3732050542 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1b0c001690 con 0x7f1b08030660 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b14e95640 10 monclient: dump: 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:35.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b16e99640 5 monclient: authenticate success, global_id 4383 2024-03-29T10:38:35.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.038+0000 7f1b16e99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:35.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.040+0000 7f1b14e95640 1 -- 172.21.15.117:0/3732050542 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b0c001860 con 0x7f1b08030660 2024-03-29T10:38:35.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.040+0000 7f1b14e95640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:35.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.040+0000 7f1b14e95640 1 -- 172.21.15.117:0/3732050542 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1b0c001c70 con 0x7f1b08030660 2024-03-29T10:38:35.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.040+0000 7f1b14e95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:35.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.040+0000 7f1b14e95640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:35.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.040+0000 7f1b14e95640 10 monclient: dump: 2024-03-29T10:38:35.027 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:35.027 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:35.027 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.040+0000 7f1b16e99640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.040+0000 7f1b16e99640 4 set_mon_vals no callback set 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.041+0000 7f1b16e99640 10 monclient: shutdown 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.041+0000 7f1b16e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.041+0000 7f1b16e99640 1 -- 172.21.15.117:0/3732050542 >> v1:172.21.15.117:6789/0 conn(0x7f1b08030660 legacy=0x7f1b080d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.042+0000 7f1b16e99640 1 -- 172.21.15.117:0/3732050542 shutdown_connections 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.042+0000 7f1b16e99640 1 -- 172.21.15.117:0/3732050542 >> 172.21.15.117:0/3732050542 conn(0x7f1b08005c00 msgr2=0x7f1b08006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:35.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.043+0000 7f1b16e99640 1 -- 172.21.15.117:0/3732050542 shutdown_connections 2024-03-29T10:38:35.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.043+0000 7f1b16e99640 1 -- 172.21.15.117:0/3732050542 wait complete. 2024-03-29T10:38:35.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.044+0000 7f1b16e99640 10 monclient: build_initial_monmap 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.044+0000 7f1b16e99640 10 monclient: monmap: 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:35.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.044+0000 7f1b16e99640 1 librados: starting msgr at 2024-03-29T10:38:35.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.044+0000 7f1b16e99640 1 librados: starting objecter 2024-03-29T10:38:35.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.044+0000 7f1b16e99640 1 Processor -- start 2024-03-29T10:38:35.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.044+0000 7f1b16e99640 1 -- start start 2024-03-29T10:38:35.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.044+0000 7f1b16e99640 1 librados: setting wanted keys 2024-03-29T10:38:35.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.044+0000 7f1b16e99640 1 librados: calling monclient init 2024-03-29T10:38:35.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.044+0000 7f1b16e99640 10 monclient: init 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 10 monclient(hunting): picked mon.noname-c con 0x7f1b08030660 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 10 monclient(hunting): picked mon.noname-a con 0x7f1b080389e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 10 monclient(hunting): picked mon.noname-b con 0x7f1b0800a910 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0800cfc0 con 0x7f1b080389e0 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b08009980 con 0x7f1b0800a910 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0800a5b0 con 0x7f1b08030660 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:35.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.046+0000 7f1b16e99640 10 monclient(hunting): authenticate will time out at 1036.308931s 2024-03-29T10:38:35.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.050+0000 7f1b16698640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1b0800a910 0x7f1b080c30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:53893/0 (socket says 172.21.15.117:53893) 2024-03-29T10:38:35.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.051+0000 7f1b16698640 1 -- 172.21.15.117:0/2461817768 learned_addr learned my addr 172.21.15.117:0/2461817768 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:35.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.051+0000 7f1b067fc640 10 client.?.objecter ms_handle_connect 0x7f1b0800a910 2024-03-29T10:38:35.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.051+0000 7f1b067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.052+0000 7f1b067fc640 10 client.?.objecter ms_handle_connect 0x7f1b080389e0 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.052+0000 7f1b067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.052+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601788964 0 0) 0x7f1b0800cfc0 con 0x7f1b080389e0 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.052+0000 7f1b067fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.052+0000 7f1b067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.052+0000 7f1b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.052+0000 7f1b067fc640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.052+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af4004eb0 con 0x7f1b080389e0 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 10 client.?.objecter ms_handle_connect 0x7f1b08030660 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:35.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166508303 0 0) 0x7f1b08009980 con 0x7f1b0800a910 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b0800cfc0 con 0x7f1b0800a910 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1550148101 0 0) 0x7f1af4004eb0 con 0x7f1b080389e0 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b08009980 con 0x7f1b080389e0 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1b000017d0 con 0x7f1b080389e0 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.053+0000 7f1b067fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 10 monclient(hunting): dump: 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:35.041 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425584682 0 0) 0x7f1b0800a5b0 con 0x7f1b08030660 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af4004eb0 con 0x7f1b08030660 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 324052923 0 0) 0x7f1b08009980 con 0x7f1b080389e0 2024-03-29T10:38:35.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.054+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 >> v1:172.21.15.117:6790/0 conn(0x7f1b08030660 legacy=0x7f1b0800d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:35.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.055+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 >> v1:172.21.15.136:6789/0 conn(0x7f1b0800a910 legacy=0x7f1b080c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:35.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.055+0000 7f1b067fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:35.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.055+0000 7f1b067fc640 1 monclient: found mon.a 2024-03-29T10:38:35.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.055+0000 7f1b067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:35.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.055+0000 7f1b067fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:35.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.055+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0802eec0 con 0x7f1b080389e0 2024-03-29T10:38:35.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.055+0000 7f1b067fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:35.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.055+0000 7f1b067fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:35.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.055+0000 7f1b067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:35.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.056+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b00001ce0 con 0x7f1b080389e0 2024-03-29T10:38:35.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.056+0000 7f1b067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.056+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1b00003790 con 0x7f1b080389e0 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.056+0000 7f1b067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b067fc640 10 monclient: dump: 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:35.045 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b07fff640 4 set_mon_vals no callback set 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b16e99640 5 monclient: authenticate success, global_id 4386 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b16e99640 10 monclient: _renew_subs 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b16e99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0802f0b0 con 0x7f1b080389e0 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b16e99640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b16e99640 10 monclient: _renew_subs 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b16e99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b080c39b0 con 0x7f1b080389e0 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.057+0000 7f1b16e99640 1 librados: init done 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.059+0000 7f1b16e99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.060+0000 7f1b16e99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:35.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.060+0000 7f1b16e99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:35.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.060+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b08007b90 con 0x7f1b080389e0 2024-03-29T10:38:35.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.061+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f1b000038f0 con 0x7f1b080389e0 2024-03-29T10:38:35.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.062+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f1b00083760 con 0x7f1b080389e0 2024-03-29T10:38:35.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.062+0000 7f1b067fc640 10 client.4386.objecter ms_dispatch 0x7f1b10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:35.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.062+0000 7f1b067fc640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:35.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.062+0000 7f1b067fc640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:35.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.062+0000 7f1b067fc640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-29T10:38:35.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.063+0000 7f1b15e97640 10 monclient: get_auth_request con 0x7f1af406d8a0 auth_method 0 2024-03-29T10:38:35.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.063+0000 7f1b067fc640 10 client.4386.objecter ms_handle_connect 0x7f1af406d8a0 2024-03-29T10:38:35.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.075+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b0004f040 con 0x7f1b080389e0 2024-03-29T10:38:35.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.075+0000 7f1b067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:35.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.075+0000 7f1b067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:35.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.418+0000 7f1b16e99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:35.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.418+0000 7f1b16e99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:35.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.418+0000 7f1b16e99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:35.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.418+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b08009b10 con 0x7f1b080389e0 2024-03-29T10:38:35.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.422+0000 7f1b067fc640 1 -- 172.21.15.117:0/2461817768 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f1b00050f70 con 0x7f1b080389e0 2024-03-29T10:38:35.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.422+0000 7f1b067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:35.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.422+0000 7f1b067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:35.406 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:38:35.406 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:35.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 10 librados: watch_flush enter 2024-03-29T10:38:35.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 10 librados: watch_flush exit 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 10 client.4386.objecter successfully canceled tick 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 >> v1:172.21.15.117:6816/376768171 conn(0x7f1af406d8a0 legacy=0x7f1af406fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 10 monclient: shutdown 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 >> v1:172.21.15.117:6789/0 conn(0x7f1b080389e0 legacy=0x7f1b08009ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 shutdown_connections 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 >> 172.21.15.117:0/2461817768 conn(0x7f1b08005c00 msgr2=0x7f1b08034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 shutdown_connections 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.426+0000 7f1b16e99640 1 -- 172.21.15.117:0/2461817768 wait complete. 2024-03-29T10:38:35.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:35.427+0000 7f1b16e99640 1 librados: shutdown 2024-03-29T10:38:35.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:42.235 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:42.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.436+0000 7fe84b59e640 10 monclient: get_monmap_and_config 2024-03-29T10:38:42.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.436+0000 7fe84b59e640 10 monclient: build_initial_monmap 2024-03-29T10:38:42.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.436+0000 7fe84b59e640 10 monclient: monmap: 2024-03-29T10:38:42.420 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:42.420 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:42.420 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:38:42.436937+0000 2024-03-29T10:38:42.420 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:38:42.436937+0000 2024-03-29T10:38:42.421 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:42.421 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:42.421 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:42.421 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:38:42.421 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:38:42.421 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:42.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.437+0000 7fe84b59e640 1 Processor -- start 2024-03-29T10:38:42.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.437+0000 7fe84b59e640 1 -- start start 2024-03-29T10:38:42.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.437+0000 7fe84b59e640 10 monclient: init 2024-03-29T10:38:42.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fe844006a70 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fe84402b100 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe844029c30 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84402a270 con 0x7fe844006a70 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84402c440 con 0x7fe844029c30 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8440c9440 con 0x7fe84402b100 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe84b59e640 10 monclient(hunting): authenticate will time out at 1043.700873s 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe849d9b640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe84402b100 0x7fe8440cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41136/0 (socket says 172.21.15.117:41136) 2024-03-29T10:38:42.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.438+0000 7fe849d9b640 1 -- 172.21.15.117:0/452025896 learned_addr learned my addr 172.21.15.117:0/452025896 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:42.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.439+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205859897 0 0) 0x7fe8440c9440 con 0x7fe84402b100 2024-03-29T10:38:42.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.439+0000 7fe84959a640 10 monclient(hunting): _negotiate 2024-03-29T10:38:42.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.439+0000 7fe84959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:42.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.439+0000 7fe84959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:42.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.439+0000 7fe84959a640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:38:42.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.439+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe834003520 con 0x7fe84402b100 2024-03-29T10:38:42.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.440+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322512058 0 0) 0x7fe84402a270 con 0x7fe844006a70 2024-03-29T10:38:42.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.440+0000 7fe84959a640 10 monclient(hunting): _negotiate 2024-03-29T10:38:42.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.440+0000 7fe84959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:42.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.440+0000 7fe84959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.440+0000 7fe84959a640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.440+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8440c9440 con 0x7fe844006a70 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.440+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1418728437 0 0) 0x7fe834003520 con 0x7fe84402b100 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.440+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 >> v1:172.21.15.136:6789/0 conn(0x7fe844029c30 legacy=0x7fe8440d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 >> v1:172.21.15.117:6789/0 conn(0x7fe844006a70 legacy=0x7fe8440c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 10 monclient: _finish_hunting 0 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 1 monclient: found mon.noname-b 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8440c95d0 con 0x7fe84402b100 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 10 monclient: _finish_auth 0 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 10 monclient: _check_auth_tickets 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe840001610 con 0x7fe84402b100 2024-03-29T10:38:42.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84959a640 10 monclient: dump: 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84b59e640 5 monclient: authenticate success, global_id 4403 2024-03-29T10:38:42.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.441+0000 7fe84b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:42.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.442+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe840001770 con 0x7fe84402b100 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.442+0000 7fe84959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.442+0000 7fe84959a640 1 -- 172.21.15.117:0/452025896 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe840001b80 con 0x7fe84402b100 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.442+0000 7fe84959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.442+0000 7fe84959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.442+0000 7fe84959a640 10 monclient: dump: 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:42.429 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.442+0000 7fe84b59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.442+0000 7fe84b59e640 4 set_mon_vals no callback set 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.443+0000 7fe84b59e640 10 monclient: shutdown 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.443+0000 7fe84b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.443+0000 7fe84b59e640 1 -- 172.21.15.117:0/452025896 >> v1:172.21.15.117:6790/0 conn(0x7fe84402b100 legacy=0x7fe8440cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.443+0000 7fe84b59e640 1 -- 172.21.15.117:0/452025896 shutdown_connections 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.443+0000 7fe84b59e640 1 -- 172.21.15.117:0/452025896 >> 172.21.15.117:0/452025896 conn(0x7fe8440058f0 msgr2=0x7fe844005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.443+0000 7fe84b59e640 1 -- 172.21.15.117:0/452025896 shutdown_connections 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.443+0000 7fe84b59e640 1 -- 172.21.15.117:0/452025896 wait complete. 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.445+0000 7fe84b59e640 10 monclient: build_initial_monmap 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.445+0000 7fe84b59e640 10 monclient: monmap: 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:42.430 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:38:42.431 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:38:42.431 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:42.431 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:42.431 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:42.431 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:38:42.431 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:38:42.431 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:42.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.445+0000 7fe84b59e640 1 librados: starting msgr at 2024-03-29T10:38:42.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.445+0000 7fe84b59e640 1 librados: starting objecter 2024-03-29T10:38:42.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.445+0000 7fe84b59e640 1 Processor -- start 2024-03-29T10:38:42.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.446+0000 7fe84b59e640 1 -- start start 2024-03-29T10:38:42.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.446+0000 7fe84b59e640 1 librados: setting wanted keys 2024-03-29T10:38:42.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.446+0000 7fe84b59e640 1 librados: calling monclient init 2024-03-29T10:38:42.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.446+0000 7fe84b59e640 10 monclient: init 2024-03-29T10:38:42.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:42.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe844006a70 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fe844029c30 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fe844155bf0 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8441561a0 con 0x7fe844155bf0 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8441574d0 con 0x7fe844029c30 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe849d9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe844029c30 0x7fe844157bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:36910/0 (socket says 172.21.15.117:36910) 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84ad9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe844155bf0 0x7fe84416b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:59834/0 (socket says 172.21.15.117:59834) 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84ad9d640 1 -- 172.21.15.117:0/545346284 learned_addr learned my addr 172.21.15.117:0/545346284 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe844156630 con 0x7fe844006a70 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 client.?.objecter ms_handle_connect 0x7fe844029c30 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:42.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 client.?.objecter ms_handle_connect 0x7fe844006a70 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe84b59e640 10 monclient(hunting): authenticate will time out at 1043.709805s 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 client.?.objecter ms_handle_connect 0x7fe844155bf0 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164526609 0 0) 0x7fe8441574d0 con 0x7fe844029c30 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.447+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe82c004eb0 con 0x7fe844029c30 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589940097 0 0) 0x7fe844156630 con 0x7fe844006a70 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:42.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8441574d0 con 0x7fe844006a70 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 811813859 0 0) 0x7fe82c004eb0 con 0x7fe844029c30 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe844156630 con 0x7fe844029c30 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe840001b40 con 0x7fe844029c30 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 10 monclient(hunting): dump: 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:42.437 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.448+0000 7fe83affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792722832 0 0) 0x7fe8441561a0 con 0x7fe844155bf0 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe82c004eb0 con 0x7fe844155bf0 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1855051833 0 0) 0x7fe8441574d0 con 0x7fe844006a70 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8441561a0 con 0x7fe844006a70 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe83c001130 con 0x7fe844006a70 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:42.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient(hunting): dump: 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1314868980 0 0) 0x7fe844156630 con 0x7fe844029c30 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 >> v1:172.21.15.117:6790/0 conn(0x7fe844006a70 legacy=0x7fe8441553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 >> v1:172.21.15.117:6789/0 conn(0x7fe844155bf0 legacy=0x7fe84416b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:42.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient: _finish_hunting 0 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 monclient: found mon.b 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8441577b0 con 0x7fe844029c30 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient: _finish_auth 0 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 10 monclient: _check_auth_tickets 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe83affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe84b59e640 5 monclient: authenticate success, global_id 4372 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe84b59e640 10 monclient: _renew_subs 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe84b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8441582e0 con 0x7fe844029c30 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe84b59e640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe84b59e640 10 monclient: _renew_subs 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe84b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:42.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe844158770 con 0x7fe844029c30 2024-03-29T10:38:42.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.449+0000 7fe84b59e640 1 librados: init done 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.450+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe840004a90 con 0x7fe844029c30 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.450+0000 7fe83affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.450+0000 7fe848d99640 4 set_mon_vals no callback set 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.451+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe840004da0 con 0x7fe844029c30 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.451+0000 7fe83affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.451+0000 7fe83affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.451+0000 7fe84b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.451+0000 7fe83affd640 10 monclient: dump: 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:42.442 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.451+0000 7fe84b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.451+0000 7fe84b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.451+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe830005080 con 0x7fe844029c30 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.452+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fe84001ad70 con 0x7fe844029c30 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.453+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fe840054620 con 0x7fe844029c30 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.453+0000 7fe83affd640 10 client.4372.objecter ms_dispatch 0x7fe84c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.453+0000 7fe83affd640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.453+0000 7fe83affd640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:42.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.453+0000 7fe84a59c640 10 monclient: get_auth_request con 0x7fe82c06e0c0 auth_method 0 2024-03-29T10:38:42.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.453+0000 7fe83affd640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-29T10:38:42.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.453+0000 7fe83affd640 10 client.4372.objecter ms_handle_connect 0x7fe82c06e0c0 2024-03-29T10:38:42.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.458+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8400508c0 con 0x7fe844029c30 2024-03-29T10:38:42.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.458+0000 7fe83affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:42.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.458+0000 7fe83affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:42.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.894+0000 7fe84b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:42.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.894+0000 7fe84b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:42.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.894+0000 7fe84b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:38:42.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.894+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe830005330 con 0x7fe844029c30 2024-03-29T10:38:42.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.894+0000 7fe83affd640 1 -- 172.21.15.117:0/545346284 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fe840050a60 con 0x7fe844029c30 2024-03-29T10:38:42.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.895+0000 7fe83affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:42.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.895+0000 7fe83affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:42.879 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:38:42.879 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.898+0000 7fe84b59e640 10 librados: watch_flush enter 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.898+0000 7fe84b59e640 10 librados: watch_flush exit 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.898+0000 7fe84b59e640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.898+0000 7fe84b59e640 10 client.4372.objecter successfully canceled tick 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.899+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 >> v1:172.21.15.117:6816/376768171 conn(0x7fe82c06e0c0 legacy=0x7fe82c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.899+0000 7fe84b59e640 10 monclient: shutdown 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.899+0000 7fe84b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.899+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 >> v1:172.21.15.136:6789/0 conn(0x7fe844029c30 legacy=0x7fe844157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.899+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 shutdown_connections 2024-03-29T10:38:42.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.899+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 >> 172.21.15.117:0/545346284 conn(0x7fe8440058f0 msgr2=0x7fe844036e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:42.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.899+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 shutdown_connections 2024-03-29T10:38:42.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.899+0000 7fe84b59e640 1 -- 172.21.15.117:0/545346284 wait complete. 2024-03-29T10:38:42.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:42.899+0000 7fe84b59e640 1 librados: shutdown 2024-03-29T10:38:42.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:49.705 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.855+0000 7f2f621ef640 10 monclient: get_monmap_and_config 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.855+0000 7f2f621ef640 10 monclient: build_initial_monmap 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.855+0000 7f2f621ef640 10 monclient: monmap: 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:38:49.855926+0000 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:38:49.855926+0000 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:49.839 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:49.840 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:38:49.840 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:38:49.840 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:49.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.855+0000 7f2f621ef640 1 Processor -- start 2024-03-29T10:38:49.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.856+0000 7f2f621ef640 1 -- start start 2024-03-29T10:38:49.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.856+0000 7f2f621ef640 10 monclient: init 2024-03-29T10:38:49.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:49.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:49.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5c1422b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:49.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 10 monclient(hunting): picked mon.noname-b con 0x7f2f5c142690 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:49.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5c0b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:49.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c142c80 con 0x7f2f5c1422b0 2024-03-29T10:38:49.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0aae40 con 0x7f2f5c0b17d0 2024-03-29T10:38:49.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c13fbd0 con 0x7f2f5c142690 2024-03-29T10:38:49.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:49.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f621ef640 10 monclient(hunting): authenticate will time out at 1051.119196s 2024-03-29T10:38:49.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f5affd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f2f5c142690 0x7f2f5c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:46528/0 (socket says 172.21.15.117:46528) 2024-03-29T10:38:49.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f5affd640 1 -- 172.21.15.117:0/1980918593 learned_addr learned my addr 172.21.15.117:0/1980918593 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:49.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088954737 0 0) 0x7f2f5c13fbd0 con 0x7f2f5c142690 2024-03-29T10:38:49.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:49.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:49.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:49.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.857+0000 7f2f5a7fc640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:38:49.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.858+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f44003520 con 0x7f2f5c142690 2024-03-29T10:38:49.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.858+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638786167 0 0) 0x7f2f5c0aae40 con 0x7f2f5c0b17d0 2024-03-29T10:38:49.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.858+0000 7f2f5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:49.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.858+0000 7f2f5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:49.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.858+0000 7f2f5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:49.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.858+0000 7f2f5a7fc640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:38:49.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.858+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c13fbd0 con 0x7f2f5c0b17d0 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.858+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3038825374 0 0) 0x7f2f5c13fbd0 con 0x7f2f5c0b17d0 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.859+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 >> v1:172.21.15.117:6790/0 conn(0x7f2f5c142690 legacy=0x7f2f5c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.859+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 >> v1:172.21.15.117:6789/0 conn(0x7f2f5c1422b0 legacy=0x7f2f5c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c151940 con 0x7f2f5c0b17d0 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2f540019f0 con 0x7f2f5c0b17d0 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:49.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f5a7fc640 10 monclient: dump: 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f621ef640 5 monclient: authenticate success, global_id 4375 2024-03-29T10:38:49.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.860+0000 7f2f621ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:49.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.861+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f54001c10 con 0x7f2f5c0b17d0 2024-03-29T10:38:49.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.861+0000 7f2f5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.861+0000 7f2f5a7fc640 1 -- 172.21.15.117:0/1980918593 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2f54001f20 con 0x7f2f5c0b17d0 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.861+0000 7f2f5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.861+0000 7f2f5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.861+0000 7f2f5a7fc640 10 monclient: dump: 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:49.857 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.862+0000 7f2f621ef640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.862+0000 7f2f621ef640 4 set_mon_vals no callback set 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.862+0000 7f2f621ef640 10 monclient: shutdown 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.862+0000 7f2f621ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.862+0000 7f2f621ef640 1 -- 172.21.15.117:0/1980918593 >> v1:172.21.15.136:6789/0 conn(0x7f2f5c0b17d0 legacy=0x7f2f5c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.863+0000 7f2f621ef640 1 -- 172.21.15.117:0/1980918593 shutdown_connections 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.863+0000 7f2f621ef640 1 -- 172.21.15.117:0/1980918593 >> 172.21.15.117:0/1980918593 conn(0x7f2f5c066a00 msgr2=0x7f2f5c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.863+0000 7f2f621ef640 1 -- 172.21.15.117:0/1980918593 shutdown_connections 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.863+0000 7f2f621ef640 1 -- 172.21.15.117:0/1980918593 wait complete. 2024-03-29T10:38:49.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.864+0000 7f2f621ef640 10 monclient: build_initial_monmap 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.864+0000 7f2f621ef640 10 monclient: monmap: 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.865+0000 7f2f621ef640 1 librados: starting msgr at 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.865+0000 7f2f621ef640 1 librados: starting objecter 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.865+0000 7f2f621ef640 1 Processor -- start 2024-03-29T10:38:49.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.865+0000 7f2f621ef640 1 -- start start 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.865+0000 7f2f621ef640 1 librados: setting wanted keys 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.865+0000 7f2f621ef640 1 librados: calling monclient init 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.865+0000 7f2f621ef640 10 monclient: init 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.866+0000 7f2f621ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.866+0000 7f2f621ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.866+0000 7f2f621ef640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5c0b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f621ef640 10 monclient(hunting): picked mon.noname-b con 0x7f2f5c1422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f621ef640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5c142690 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f621ef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0acb60 con 0x7f2f5c0b17d0 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f621ef640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0ace40 con 0x7f2f5c1422b0 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f621ef640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0ad120 con 0x7f2f5c142690 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f621ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f621ef640 10 monclient(hunting): authenticate will time out at 1051.130381s 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f5bfff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f2f5c142690 0x7f2f5c12fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:47600/0 (socket says 172.21.15.117:47600) 2024-03-29T10:38:49.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f5bfff640 1 -- 172.21.15.117:0/418085547 learned_addr learned my addr 172.21.15.117:0/418085547 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f43fff640 10 client.?.objecter ms_handle_connect 0x7f2f5c142690 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f43fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920243721 0 0) 0x7f2f5c0ad120 con 0x7f2f5c142690 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f43fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f43fff640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.868+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f48004eb0 con 0x7f2f5c142690 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.869+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 13483375 0 0) 0x7f2f48004eb0 con 0x7f2f5c142690 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.869+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f5c0ad120 con 0x7f2f5c142690 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.869+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2f540019b0 con 0x7f2f5c142690 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.869+0000 7f2f43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.869+0000 7f2f43fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.869+0000 7f2f43fff640 10 monclient(hunting): dump: 2024-03-29T10:38:49.861 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.869+0000 7f2f43fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.869+0000 7f2f43fff640 10 client.?.objecter ms_handle_connect 0x7f2f5c1422b0 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.869+0000 7f2f43fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 client.?.objecter ms_handle_connect 0x7f2f5c0b17d0 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:49.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4260781873 0 0) 0x7f2f5c0ad120 con 0x7f2f5c142690 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 >> v1:172.21.15.136:6789/0 conn(0x7f2f5c1422b0 legacy=0x7f2f5c0b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 >> v1:172.21.15.117:6789/0 conn(0x7f2f5c0b17d0 legacy=0x7f2f5c0ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 1 monclient: found mon.c 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c132830 con 0x7f2f5c142690 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 monclient: _finish_auth 0 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f621ef640 5 monclient: authenticate success, global_id 4412 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f621ef640 10 monclient: _renew_subs 2024-03-29T10:38:49.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f621ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:49.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5c132a80 con 0x7f2f5c142690 2024-03-29T10:38:49.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f621ef640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:49.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f621ef640 10 monclient: _renew_subs 2024-03-29T10:38:49.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f621ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:49.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5c0b1190 con 0x7f2f5c142690 2024-03-29T10:38:49.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f54003c20 con 0x7f2f5c142690 2024-03-29T10:38:49.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:49.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2f54003fb0 con 0x7f2f5c142690 2024-03-29T10:38:49.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:49.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f43fff640 10 monclient: dump: 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:49.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.870+0000 7f2f621ef640 1 librados: init done 2024-03-29T10:38:49.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.871+0000 7f2f59ffb640 4 set_mon_vals no callback set 2024-03-29T10:38:49.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.872+0000 7f2f621ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:49.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.872+0000 7f2f621ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:49.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.872+0000 7f2f621ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:49.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.872+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c005080 con 0x7f2f5c142690 2024-03-29T10:38:49.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.873+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f2f54004110 con 0x7f2f5c142690 2024-03-29T10:38:49.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.875+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f2f54084f70 con 0x7f2f5c142690 2024-03-29T10:38:49.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.875+0000 7f2f43fff640 10 client.4412.objecter ms_dispatch 0x7f2f5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:49.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.875+0000 7f2f43fff640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:49.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.875+0000 7f2f43fff640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:49.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.875+0000 7f2f43fff640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T10:38:49.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.875+0000 7f2f5b7fe640 10 monclient: get_auth_request con 0x7f2f4806dbf0 auth_method 0 2024-03-29T10:38:49.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.875+0000 7f2f43fff640 10 client.4412.objecter ms_handle_connect 0x7f2f4806dbf0 2024-03-29T10:38:49.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.882+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f540508b0 con 0x7f2f5c142690 2024-03-29T10:38:49.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.882+0000 7f2f43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:49.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:49.882+0000 7f2f43fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:50.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.211+0000 7f2f621ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:50.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.211+0000 7f2f621ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:50.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.211+0000 7f2f621ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:38:50.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.211+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f3c005330 con 0x7f2f5c142690 2024-03-29T10:38:50.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.212+0000 7f2f43fff640 1 -- 172.21.15.117:0/418085547 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f2f54084620 con 0x7f2f5c142690 2024-03-29T10:38:50.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.212+0000 7f2f43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:50.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.212+0000 7f2f43fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:50.196 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:38:50.197 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 10 librados: watch_flush enter 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 10 librados: watch_flush exit 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 10 client.4412.objecter successfully canceled tick 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 >> v1:172.21.15.117:6816/376768171 conn(0x7f2f4806dbf0 legacy=0x7f2f480700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 10 monclient: shutdown 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 >> v1:172.21.15.117:6790/0 conn(0x7f2f5c142690 legacy=0x7f2f5c12fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 shutdown_connections 2024-03-29T10:38:50.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.215+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 >> 172.21.15.117:0/418085547 conn(0x7f2f5c066a00 msgr2=0x7f2f5c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:50.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.216+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 shutdown_connections 2024-03-29T10:38:50.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.216+0000 7f2f621ef640 1 -- 172.21.15.117:0/418085547 wait complete. 2024-03-29T10:38:50.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:50.216+0000 7f2f621ef640 1 librados: shutdown 2024-03-29T10:38:50.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:57.022 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:57.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.191+0000 7fbad1b58640 10 monclient: get_monmap_and_config 2024-03-29T10:38:57.176 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.191+0000 7fbad1b58640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.191+0000 7fbad1b58640 10 monclient: monmap: 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:38:57.192507+0000 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:38:57.192507+0000 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.192+0000 7fbad1b58640 1 Processor -- start 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.192+0000 7fbad1b58640 1 -- start start 2024-03-29T10:38:57.177 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.192+0000 7fbad1b58640 10 monclient: init 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 10 monclient(hunting): picked mon.noname-a con 0x7fbacc0a9ac0 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 10 monclient(hunting): picked mon.noname-c con 0x7fbacc13ee90 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 10 monclient(hunting): picked mon.noname-b con 0x7fbacc0af900 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc0afd10 con 0x7fbacc0a9ac0 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc0af6f0 con 0x7fbacc13ee90 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc0aac80 con 0x7fbacc0af900 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.193+0000 7fbad1b58640 10 monclient(hunting): authenticate will time out at 1058.455746s 2024-03-29T10:38:57.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.196+0000 7fbacbfff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fbacc0af900 0x7fbacc149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:51543/0 (socket says 172.21.15.117:51543) 2024-03-29T10:38:57.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.196+0000 7fbacbfff640 1 -- 172.21.15.117:0/2874777 learned_addr learned my addr 172.21.15.117:0/2874777 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:57.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.199+0000 7fbaca7fc640 1 -- 172.21.15.117:0/2874777 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998164971 0 0) 0x7fbacc0aac80 con 0x7fbacc0af900 2024-03-29T10:38:57.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.199+0000 7fbaca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.199+0000 7fbaca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.199+0000 7fbaca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.199+0000 7fbaca7fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:38:57.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.199+0000 7fbaca7fc640 1 -- 172.21.15.117:0/2874777 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab4003520 con 0x7fbacc0af900 2024-03-29T10:38:57.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 1 -- 172.21.15.117:0/2874777 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2972154992 0 0) 0x7fbab4003520 con 0x7fbacc0af900 2024-03-29T10:38:57.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 1 -- 172.21.15.117:0/2874777 >> v1:172.21.15.136:6789/0 conn(0x7fbacc13ee90 legacy=0x7fbacc13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:57.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 1 -- 172.21.15.117:0/2874777 >> v1:172.21.15.117:6789/0 conn(0x7fbacc0a9ac0 legacy=0x7fbacc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:57.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 1 monclient: found mon.noname-b 2024-03-29T10:38:57.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 1 -- 172.21.15.117:0/2874777 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc0aaf60 con 0x7fbacc0af900 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 1 -- 172.21.15.117:0/2874777 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbac4001a10 con 0x7fbacc0af900 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 10 monclient: dump: 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:57.191 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbad1b58640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 1 -- 172.21.15.117:0/2874777 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac4001be0 con 0x7fbacc0af900 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbaca7fc640 1 -- 172.21.15.117:0/2874777 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbac4002030 con 0x7fbacc0af900 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbad1b58640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.200+0000 7fbad1b58640 4 set_mon_vals no callback set 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.202+0000 7fbad1b58640 10 monclient: shutdown 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.202+0000 7fbad1b58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.192 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.202+0000 7fbad1b58640 1 -- 172.21.15.117:0/2874777 >> v1:172.21.15.117:6790/0 conn(0x7fbacc0af900 legacy=0x7fbacc149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.204+0000 7fbad1b58640 1 -- 172.21.15.117:0/2874777 shutdown_connections 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.204+0000 7fbad1b58640 1 -- 172.21.15.117:0/2874777 >> 172.21.15.117:0/2874777 conn(0x7fbacc0b35e0 msgr2=0x7fbacc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.204+0000 7fbaca7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.204+0000 7fbad1b58640 1 -- 172.21.15.117:0/2874777 shutdown_connections 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.204+0000 7fbad1b58640 1 -- 172.21.15.117:0/2874777 wait complete. 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.205+0000 7fbad1b58640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.205+0000 7fbad1b58640 10 monclient: monmap: 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:38:57.193 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:38:57.194 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:57.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.206+0000 7fbad1b58640 1 librados: starting msgr at 2024-03-29T10:38:57.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.206+0000 7fbad1b58640 1 librados: starting objecter 2024-03-29T10:38:57.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.206+0000 7fbad1b58640 1 Processor -- start 2024-03-29T10:38:57.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.206+0000 7fbad1b58640 1 -- start start 2024-03-29T10:38:57.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.206+0000 7fbad1b58640 1 librados: setting wanted keys 2024-03-29T10:38:57.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.206+0000 7fbad1b58640 1 librados: calling monclient init 2024-03-29T10:38:57.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.206+0000 7fbad1b58640 10 monclient: init 2024-03-29T10:38:57.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:38:57.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 10 monclient(hunting): picked mon.noname-b con 0x7fbacc0a9ac0 addr v1:172.21.15.136:6789/0 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 10 monclient(hunting): picked mon.noname-c con 0x7fbacc13ee90 addr v1:172.21.15.117:6790/0 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 10 monclient(hunting): picked mon.noname-a con 0x7fbacc140460 addr v1:172.21.15.117:6789/0 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc13fb30 con 0x7fbacc140460 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc141ad0 con 0x7fbacc0a9ac0 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbacc141db0 con 0x7fbacc13ee90 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.207+0000 7fbad1b58640 10 monclient(hunting): authenticate will time out at 1058.469768s 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.210+0000 7fbacbfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fbacc140460 0x7fbacc1e1cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:41919/0 (socket says 172.21.15.117:41919) 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.210+0000 7fbacbfff640 1 -- 172.21.15.117:0/2914705872 learned_addr learned my addr 172.21.15.117:0/2914705872 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.210+0000 7fbab3fff640 10 client.?.objecter ms_handle_connect 0x7fbacc140460 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.210+0000 7fbab3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864072306 0 0) 0x7fbacc13fb30 con 0x7fbacc140460 2024-03-29T10:38:57.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab8004eb0 con 0x7fbacc140460 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053078511 0 0) 0x7fbab8004eb0 con 0x7fbacc140460 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbacc13fb30 con 0x7fbacc140460 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbac4001fb0 con 0x7fbacc140460 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 10 monclient(hunting): dump: 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:57.203 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.211+0000 7fbab3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1252582690 0 0) 0x7fbacc13fb30 con 0x7fbacc140460 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 >> v1:172.21.15.117:6790/0 conn(0x7fbacc13ee90 legacy=0x7fbacc13c8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 >> v1:172.21.15.136:6789/0 conn(0x7fbacc0a9ac0 legacy=0x7fbacc13c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 1 monclient: found mon.a 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:57.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc13f6e0 con 0x7fbacc140460 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbab3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbad1b58640 5 monclient: authenticate success, global_id 4398 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbad1b58640 10 monclient: _renew_subs 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbad1b58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbacc140b20 con 0x7fbacc140460 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbad1b58640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbad1b58640 10 monclient: _renew_subs 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbad1b58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc1e35b0 con 0x7fbacc140460 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.212+0000 7fbad1b58640 1 librados: init done 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbad1b58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbad1b58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbad1b58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:57.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc075f10 con 0x7fbacc140460 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac40015f0 con 0x7fbacc140460 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbab3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbac4005070 con 0x7fbacc140460 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbab3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbab3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbab3fff640 10 monclient: dump: 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:38:57.206 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.215+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fbac4005290 con 0x7fbacc140460 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.214+0000 7fbac97fa640 4 set_mon_vals no callback set 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.216+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fbac4084f70 con 0x7fbacc140460 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.216+0000 7fbab3fff640 10 client.4398.objecter ms_dispatch 0x7fbacc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.216+0000 7fbab3fff640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.216+0000 7fbab3fff640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.216+0000 7fbab3fff640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.216+0000 7fbacb7fe640 10 monclient: get_auth_request con 0x7fbab806dce0 auth_method 0 2024-03-29T10:38:57.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.217+0000 7fbab3fff640 10 client.4398.objecter ms_handle_connect 0x7fbab806dce0 2024-03-29T10:38:57.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.225+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbac4050850 con 0x7fbacc140460 2024-03-29T10:38:57.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.225+0000 7fbab3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.225+0000 7fbab3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:57.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.538+0000 7fbad1b58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:57.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.539+0000 7fbad1b58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:57.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.539+0000 7fbad1b58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:38:57.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.539+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbacc14d120 con 0x7fbacc140460 2024-03-29T10:38:57.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.540+0000 7fbab3fff640 1 -- 172.21.15.117:0/2914705872 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fbac4052780 con 0x7fbacc140460 2024-03-29T10:38:57.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.540+0000 7fbab3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:57.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.540+0000 7fbab3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:57.524 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:38:57.524 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:57.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.544+0000 7fbad1b58640 10 librados: watch_flush enter 2024-03-29T10:38:57.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.544+0000 7fbad1b58640 10 librados: watch_flush exit 2024-03-29T10:38:57.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.544+0000 7fbad1b58640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T10:38:57.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.544+0000 7fbad1b58640 10 client.4398.objecter successfully canceled tick 2024-03-29T10:38:57.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.544+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 >> v1:172.21.15.117:6816/376768171 conn(0x7fbab806dce0 legacy=0x7fbab80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.530 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.546+0000 7fbad1b58640 10 monclient: shutdown 2024-03-29T10:38:57.530 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.546+0000 7fbad1b58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.530 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.546+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 >> v1:172.21.15.117:6789/0 conn(0x7fbacc140460 legacy=0x7fbacc1e1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.547+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 shutdown_connections 2024-03-29T10:38:57.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.547+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 >> 172.21.15.117:0/2914705872 conn(0x7fbacc0b35e0 msgr2=0x7fbacc0b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.547+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 shutdown_connections 2024-03-29T10:38:57.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.547+0000 7fbad1b58640 1 -- 172.21.15.117:0/2914705872 wait complete. 2024-03-29T10:38:57.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:38:57.547+0000 7fbad1b58640 1 librados: shutdown 2024-03-29T10:38:57.552 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:04.359 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:04.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.519+0000 7fa41c6f6640 10 monclient: get_monmap_and_config 2024-03-29T10:39:04.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.519+0000 7fa41c6f6640 10 monclient: build_initial_monmap 2024-03-29T10:39:04.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.519+0000 7fa41c6f6640 10 monclient: monmap: 2024-03-29T10:39:04.506 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:04.506 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:04.506 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:39:04.520761+0000 2024-03-29T10:39:04.506 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:39:04.520761+0000 2024-03-29T10:39:04.506 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:04.506 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:04.507 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:04.507 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:39:04.507 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:39:04.507 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:04.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.521+0000 7fa41c6f6640 1 Processor -- start 2024-03-29T10:39:04.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.522+0000 7fa41c6f6640 1 -- start start 2024-03-29T10:39:04.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.522+0000 7fa41c6f6640 10 monclient: init 2024-03-29T10:39:04.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:04.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:04.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa4140a9ac0 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:04.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa41413ee90 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:04.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa4140af900 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:04.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4140afd10 con 0x7fa41413ee90 2024-03-29T10:39:04.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4140af6f0 con 0x7fa4140a9ac0 2024-03-29T10:39:04.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4140aac80 con 0x7fa4140af900 2024-03-29T10:39:04.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:04.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41c6f6640 10 monclient(hunting): authenticate will time out at 1065.785314s 2024-03-29T10:39:04.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41ac93640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fa4140af900 0x7fa414149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:48444/0 (socket says 172.21.15.117:48444) 2024-03-29T10:39:04.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.523+0000 7fa41ac93640 1 -- 172.21.15.117:0/554458365 learned_addr learned my addr 172.21.15.117:0/554458365 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:04.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.524+0000 7fa419490640 1 -- 172.21.15.117:0/554458365 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129376330 0 0) 0x7fa4140aac80 con 0x7fa4140af900 2024-03-29T10:39:04.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.524+0000 7fa419490640 10 monclient(hunting): _negotiate 2024-03-29T10:39:04.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.524+0000 7fa419490640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:04.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.524+0000 7fa419490640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:04.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.524+0000 7fa419490640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:39:04.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.524+0000 7fa419490640 1 -- 172.21.15.117:0/554458365 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa400003520 con 0x7fa4140af900 2024-03-29T10:39:04.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.524+0000 7fa419490640 1 -- 172.21.15.117:0/554458365 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1384699735 0 0) 0x7fa400003520 con 0x7fa4140af900 2024-03-29T10:39:04.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.524+0000 7fa419490640 1 -- 172.21.15.117:0/554458365 >> v1:172.21.15.136:6789/0 conn(0x7fa4140a9ac0 legacy=0x7fa4140adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:04.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.528+0000 7fa419490640 1 -- 172.21.15.117:0/554458365 >> v1:172.21.15.117:6789/0 conn(0x7fa41413ee90 legacy=0x7fa41413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:04.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 10 monclient: _finish_hunting 0 2024-03-29T10:39:04.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 1 monclient: found mon.noname-b 2024-03-29T10:39:04.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 1 -- 172.21.15.117:0/554458365 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4140aaf60 con 0x7fa4140af900 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 10 monclient: _finish_auth 0 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 10 monclient: _check_auth_tickets 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 1 -- 172.21.15.117:0/554458365 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fa40c001a10 con 0x7fa4140af900 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.529+0000 7fa419490640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa419490640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa419490640 10 monclient: dump: 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:04.516 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa419490640 1 -- 172.21.15.117:0/554458365 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa40c001be0 con 0x7fa4140af900 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa419490640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa419490640 1 -- 172.21.15.117:0/554458365 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fa40c002030 con 0x7fa4140af900 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa419490640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa419490640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa419490640 10 monclient: dump: 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:04.517 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa41c6f6640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa41c6f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.530+0000 7fa41c6f6640 4 set_mon_vals no callback set 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.531+0000 7fa41c6f6640 10 monclient: shutdown 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.531+0000 7fa41c6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:04.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.531+0000 7fa41c6f6640 1 -- 172.21.15.117:0/554458365 >> v1:172.21.15.117:6790/0 conn(0x7fa4140af900 legacy=0x7fa414149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:04.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.532+0000 7fa41c6f6640 1 -- 172.21.15.117:0/554458365 shutdown_connections 2024-03-29T10:39:04.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.532+0000 7fa41c6f6640 1 -- 172.21.15.117:0/554458365 >> 172.21.15.117:0/554458365 conn(0x7fa4140b35e0 msgr2=0x7fa4140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.532+0000 7fa41c6f6640 1 -- 172.21.15.117:0/554458365 shutdown_connections 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.532+0000 7fa41c6f6640 1 -- 172.21.15.117:0/554458365 wait complete. 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.533+0000 7fa41c6f6640 10 monclient: build_initial_monmap 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.533+0000 7fa41c6f6640 10 monclient: monmap: 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:04.522 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.534+0000 7fa41c6f6640 1 librados: starting msgr at 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.534+0000 7fa41c6f6640 1 librados: starting objecter 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.534+0000 7fa41c6f6640 1 Processor -- start 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.534+0000 7fa41c6f6640 1 -- start start 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.534+0000 7fa41c6f6640 1 librados: setting wanted keys 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.534+0000 7fa41c6f6640 1 librados: calling monclient init 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.534+0000 7fa41c6f6640 10 monclient: init 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.535+0000 7fa41c6f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.535+0000 7fa41c6f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.535+0000 7fa41c6f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa4140a9ac0 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.535+0000 7fa41c6f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa4140af900 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.536+0000 7fa41c6f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa41413ee90 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:04.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.536+0000 7fa41c6f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa414141410 con 0x7fa4140af900 2024-03-29T10:39:04.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.536+0000 7fa41c6f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4141416f0 con 0x7fa4140a9ac0 2024-03-29T10:39:04.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.536+0000 7fa41c6f6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa414142220 con 0x7fa41413ee90 2024-03-29T10:39:04.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.536+0000 7fa41c6f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:04.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.536+0000 7fa41c6f6640 10 monclient(hunting): authenticate will time out at 1065.798805s 2024-03-29T10:39:04.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.537+0000 7fa41ac93640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fa41413ee90 0x7fa4141e6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40468/0 (socket says 172.21.15.117:40468) 2024-03-29T10:39:04.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.537+0000 7fa41ac93640 1 -- 172.21.15.117:0/2711714301 learned_addr learned my addr 172.21.15.117:0/2711714301 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:04.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.537+0000 7fa40affd640 10 client.?.objecter ms_handle_connect 0x7fa41413ee90 2024-03-29T10:39:04.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.537+0000 7fa40affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:04.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 client.?.objecter ms_handle_connect 0x7fa4140af900 2024-03-29T10:39:04.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:04.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 client.?.objecter ms_handle_connect 0x7fa4140a9ac0 2024-03-29T10:39:04.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:04.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826493572 0 0) 0x7fa414142220 con 0x7fa41413ee90 2024-03-29T10:39:04.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f8004eb0 con 0x7fa41413ee90 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997447647 0 0) 0x7fa4141416f0 con 0x7fa4140a9ac0 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:39:04.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa414142220 con 0x7fa4140a9ac0 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166549705 0 0) 0x7fa414141410 con 0x7fa4140af900 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4141416f0 con 0x7fa4140af900 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3633794342 0 0) 0x7fa3f8004eb0 con 0x7fa41413ee90 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa414141410 con 0x7fa41413ee90 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fa40c001ef0 con 0x7fa41413ee90 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:04.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): dump: 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:04.534 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.540+0000 7fa40affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2801805918 0 0) 0x7fa414141410 con 0x7fa41413ee90 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 >> v1:172.21.15.136:6789/0 conn(0x7fa4140a9ac0 legacy=0x7fa414141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 >> v1:172.21.15.117:6789/0 conn(0x7fa4140af900 legacy=0x7fa4141df060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 1 monclient: found mon.c 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa414140570 con 0x7fa41413ee90 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 10 monclient: _finish_auth 0 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.541+0000 7fa40affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.542+0000 7fa41c6f6640 5 monclient: authenticate success, global_id 4421 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.542+0000 7fa41c6f6640 10 monclient: _renew_subs 2024-03-29T10:39:04.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.542+0000 7fa41c6f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.542+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4140b2f90 con 0x7fa41413ee90 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.542+0000 7fa41c6f6640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.542+0000 7fa41c6f6640 10 monclient: _renew_subs 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.542+0000 7fa41c6f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.542+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa414138eb0 con 0x7fa41413ee90 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.542+0000 7fa41c6f6640 1 librados: init done 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.543+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa40c002090 con 0x7fa41413ee90 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.543+0000 7fa40affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.543+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fa40c0050e0 con 0x7fa41413ee90 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.543+0000 7fa40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.543+0000 7fa418c8f640 4 set_mon_vals no callback set 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.544+0000 7fa41c6f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.544+0000 7fa40affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.544+0000 7fa40affd640 10 monclient: dump: 2024-03-29T10:39:04.536 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.544+0000 7fa41c6f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.544+0000 7fa41c6f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.544+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc005080 con 0x7fa41413ee90 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.545+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fa40c01b0b0 con 0x7fa41413ee90 2024-03-29T10:39:04.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.546+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fa40c084e00 con 0x7fa41413ee90 2024-03-29T10:39:04.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.546+0000 7fa40affd640 10 client.4421.objecter ms_dispatch 0x7fa414000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:04.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.546+0000 7fa40affd640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:04.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.546+0000 7fa40affd640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:04.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.546+0000 7fa41a492640 10 monclient: get_auth_request con 0x7fa3f806dde0 auth_method 0 2024-03-29T10:39:04.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.547+0000 7fa40affd640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T10:39:04.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.547+0000 7fa40affd640 10 client.4421.objecter ms_handle_connect 0x7fa3f806dde0 2024-03-29T10:39:04.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.553+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa40c0506e0 con 0x7fa41413ee90 2024-03-29T10:39:04.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.553+0000 7fa40affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:04.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.553+0000 7fa40affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:04.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.884+0000 7fa41c6f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:04.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.884+0000 7fa41c6f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:04.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.884+0000 7fa41c6f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:04.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.884+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3fc005330 con 0x7fa41413ee90 2024-03-29T10:39:04.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.885+0000 7fa40affd640 1 -- 172.21.15.117:0/2711714301 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fa40c052610 con 0x7fa41413ee90 2024-03-29T10:39:04.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.885+0000 7fa40affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:04.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.885+0000 7fa40affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:04.872 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:39:04.872 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:04.873 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.888+0000 7fa41c6f6640 10 librados: watch_flush enter 2024-03-29T10:39:04.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.888+0000 7fa41c6f6640 10 librados: watch_flush exit 2024-03-29T10:39:04.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.889+0000 7fa41c6f6640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T10:39:04.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.889+0000 7fa41c6f6640 10 client.4421.objecter successfully canceled tick 2024-03-29T10:39:04.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.889+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 >> v1:172.21.15.117:6816/376768171 conn(0x7fa3f806dde0 legacy=0x7fa3f80702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:04.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.889+0000 7fa41c6f6640 10 monclient: shutdown 2024-03-29T10:39:04.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.889+0000 7fa41c6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:04.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.889+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 >> v1:172.21.15.117:6790/0 conn(0x7fa41413ee90 legacy=0x7fa4141e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:04.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.889+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 shutdown_connections 2024-03-29T10:39:04.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.889+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 >> 172.21.15.117:0/2711714301 conn(0x7fa4140b35e0 msgr2=0x7fa41413b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:04.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.891+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 shutdown_connections 2024-03-29T10:39:04.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.891+0000 7fa41c6f6640 1 -- 172.21.15.117:0/2711714301 wait complete. 2024-03-29T10:39:04.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:04.891+0000 7fa41c6f6640 1 librados: shutdown 2024-03-29T10:39:04.896 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:11.703 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.863+0000 7f43148f6640 10 monclient: get_monmap_and_config 2024-03-29T10:39:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.863+0000 7f43148f6640 10 monclient: build_initial_monmap 2024-03-29T10:39:11.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.863+0000 7f43148f6640 10 monclient: monmap: 2024-03-29T10:39:11.849 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:11.849 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:11.850 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:39:11.865647+0000 2024-03-29T10:39:11.850 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:39:11.865647+0000 2024-03-29T10:39:11.850 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:11.850 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:11.850 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:11.850 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:39:11.850 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:39:11.850 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:11.853 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.867+0000 7f43148f6640 1 Processor -- start 2024-03-29T10:39:11.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.868+0000 7f43148f6640 1 -- start start 2024-03-29T10:39:11.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.868+0000 7f43148f6640 10 monclient: init 2024-03-29T10:39:11.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.869+0000 7f43148f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.870+0000 7f43148f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.870+0000 7f43148f6640 10 monclient(hunting): picked mon.noname-a con 0x7f43101422b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.872+0000 7f43148f6640 10 monclient(hunting): picked mon.noname-c con 0x7f4310142690 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.872+0000 7f43148f6640 10 monclient(hunting): picked mon.noname-b con 0x7f43100b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.872+0000 7f43148f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4310142c80 con 0x7f43101422b0 2024-03-29T10:39:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.872+0000 7f43148f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100aae40 con 0x7f4310142690 2024-03-29T10:39:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.872+0000 7f43148f6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431013fbd0 con 0x7f43100b17d0 2024-03-29T10:39:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.872+0000 7f43148f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:11.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.872+0000 7f43148f6640 10 monclient(hunting): authenticate will time out at 1073.135773s 2024-03-29T10:39:11.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430e59c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f43101422b0 0x7f431013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:34549/0 (socket says 172.21.15.117:34549) 2024-03-29T10:39:11.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430e59c640 1 -- 172.21.15.117:0/3856654485 learned_addr learned my addr 172.21.15.117:0/3856654485 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:11.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430dd9b640 1 -- 172.21.15.117:0/3856654485 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504271804 0 0) 0x7f4310142c80 con 0x7f43101422b0 2024-03-29T10:39:11.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:39:11.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:11.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:11.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430dd9b640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:39:11.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430dd9b640 1 -- 172.21.15.117:0/3856654485 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42fc003520 con 0x7f43101422b0 2024-03-29T10:39:11.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430dd9b640 1 -- 172.21.15.117:0/3856654485 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347733636 0 0) 0x7f42fc003520 con 0x7f43101422b0 2024-03-29T10:39:11.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.875+0000 7f430dd9b640 1 -- 172.21.15.117:0/3856654485 >> v1:172.21.15.117:6790/0 conn(0x7f43100b17d0 legacy=0x7f43100b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 1 -- 172.21.15.117:0/3856654485 >> v1:172.21.15.136:6789/0 conn(0x7f4310142690 legacy=0x7f43100b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 1 monclient: found mon.noname-a 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 1 -- 172.21.15.117:0/3856654485 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4310151940 con 0x7f43101422b0 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 10 monclient: _finish_auth 0 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 1 -- 172.21.15.117:0/3856654485 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4300001690 con 0x7f43101422b0 2024-03-29T10:39:11.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.876+0000 7f430dd9b640 10 monclient: dump: 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.877+0000 7f430dd9b640 1 -- 172.21.15.117:0/3856654485 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43000017f0 con 0x7f43101422b0 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.877+0000 7f430dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:11.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.877+0000 7f430dd9b640 1 -- 172.21.15.117:0/3856654485 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4300001c40 con 0x7f43101422b0 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.877+0000 7f430dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.877+0000 7f430dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.877+0000 7f430dd9b640 10 monclient: dump: 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.877+0000 7f43148f6640 5 monclient: authenticate success, global_id 4404 2024-03-29T10:39:11.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.877+0000 7f43148f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.877+0000 7f43148f6640 4 set_mon_vals no callback set 2024-03-29T10:39:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.878+0000 7f43148f6640 10 monclient: shutdown 2024-03-29T10:39:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.878+0000 7f43148f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.879+0000 7f43148f6640 1 -- 172.21.15.117:0/3856654485 >> v1:172.21.15.117:6789/0 conn(0x7f43101422b0 legacy=0x7f431013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.879+0000 7f43148f6640 1 -- 172.21.15.117:0/3856654485 shutdown_connections 2024-03-29T10:39:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.879+0000 7f43148f6640 1 -- 172.21.15.117:0/3856654485 >> 172.21.15.117:0/3856654485 conn(0x7f4310066a00 msgr2=0x7f4310066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.880+0000 7f43148f6640 1 -- 172.21.15.117:0/3856654485 shutdown_connections 2024-03-29T10:39:11.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.881+0000 7f43148f6640 1 -- 172.21.15.117:0/3856654485 wait complete. 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.882+0000 7f43148f6640 10 monclient: build_initial_monmap 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.882+0000 7f43148f6640 10 monclient: monmap: 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:39:11.868 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:11.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.883+0000 7f43148f6640 1 librados: starting msgr at 2024-03-29T10:39:11.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.883+0000 7f43148f6640 1 librados: starting objecter 2024-03-29T10:39:11.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.883+0000 7f43148f6640 1 Processor -- start 2024-03-29T10:39:11.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.883+0000 7f43148f6640 1 -- start start 2024-03-29T10:39:11.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.883+0000 7f43148f6640 1 librados: setting wanted keys 2024-03-29T10:39:11.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.883+0000 7f43148f6640 1 librados: calling monclient init 2024-03-29T10:39:11.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.884+0000 7f43148f6640 10 monclient: init 2024-03-29T10:39:11.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.884+0000 7f43148f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:11.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.884+0000 7f43148f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:11.873 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.887+0000 7f43148f6640 10 monclient(hunting): picked mon.noname-b con 0x7f43100b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:11.873 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.888+0000 7f43148f6640 10 monclient(hunting): picked mon.noname-c con 0x7f4310142690 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:11.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.891+0000 7f430e59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f43100b17d0 0x7f43100af0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:55503/0 (socket says 172.21.15.117:55503) 2024-03-29T10:39:11.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.892+0000 7f430e59c640 1 -- 172.21.15.117:0/332581725 learned_addr learned my addr 172.21.15.117:0/332581725 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:11.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.892+0000 7f43148f6640 10 monclient(hunting): picked mon.noname-a con 0x7f43100afde0 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:11.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.892+0000 7f4306ffd640 10 client.?.objecter ms_handle_connect 0x7f43100b17d0 2024-03-29T10:39:11.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.892+0000 7f4306ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:11.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.892+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100b03a0 con 0x7f43100afde0 2024-03-29T10:39:11.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.892+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100b0f20 con 0x7f43100b17d0 2024-03-29T10:39:11.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.892+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100b1200 con 0x7f4310142690 2024-03-29T10:39:11.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.892+0000 7f43148f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:11.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.892+0000 7f43148f6640 10 monclient(hunting): authenticate will time out at 1073.155695s 2024-03-29T10:39:11.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951393666 0 0) 0x7f43100b0f20 con 0x7f43100b17d0 2024-03-29T10:39:11.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:11.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:11.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:11.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:39:11.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f0004eb0 con 0x7f43100b17d0 2024-03-29T10:39:11.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 client.?.objecter ms_handle_connect 0x7f4310142690 2024-03-29T10:39:11.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1939587609 0 0) 0x7f42f0004eb0 con 0x7f43100b17d0 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43100b0f20 con 0x7f43100b17d0 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4300001bc0 con 0x7f43100b17d0 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:11.882 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.893+0000 7f4306ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309234393 0 0) 0x7f43100b1200 con 0x7f4310142690 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f0004eb0 con 0x7f4310142690 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 10 client.?.objecter ms_handle_connect 0x7f43100afde0 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1966657397 0 0) 0x7f42f0004eb0 con 0x7f4310142690 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43100b1200 con 0x7f4310142690 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f42f8001740 con 0x7f4310142690 2024-03-29T10:39:11.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.894+0000 7f4306ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805813486 0 0) 0x7f43100b03a0 con 0x7f43100afde0 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:11.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f0004eb0 con 0x7f43100afde0 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 51856102 0 0) 0x7f43100b0f20 con 0x7f43100b17d0 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 >> v1:172.21.15.117:6790/0 conn(0x7f4310142690 legacy=0x7f43101dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 >> v1:172.21.15.117:6789/0 conn(0x7f43100afde0 legacy=0x7f43101e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 1 monclient: found mon.b 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43101db280 con 0x7f43100b17d0 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f43148f6640 5 monclient: authenticate success, global_id 4384 2024-03-29T10:39:11.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f43148f6640 10 monclient: _renew_subs 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f43148f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43101db470 con 0x7f43100b17d0 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4300004a90 con 0x7f43100b17d0 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f4300004ee0 con 0x7f43100b17d0 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f4306ffd640 10 monclient: dump: 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.886 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f43148f6640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f43148f6640 10 monclient: _renew_subs 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f43148f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f431006de90 con 0x7f43100b17d0 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.895+0000 7f43148f6640 1 librados: init done 2024-03-29T10:39:11.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.896+0000 7f430d59a640 4 set_mon_vals no callback set 2024-03-29T10:39:11.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.897+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f4300005040 con 0x7f43100b17d0 2024-03-29T10:39:11.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.898+0000 7f4307fff640 10 monclient: get_auth_request con 0x7f42f0076750 auth_method 0 2024-03-29T10:39:11.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.898+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f43000549f0 con 0x7f43100b17d0 2024-03-29T10:39:11.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.898+0000 7f4306ffd640 10 client.4384.objecter ms_dispatch 0x7f4310000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:11.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.898+0000 7f4306ffd640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:11.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.898+0000 7f4306ffd640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:11.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.898+0000 7f4306ffd640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T10:39:11.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.898+0000 7f4306ffd640 10 client.4384.objecter ms_handle_connect 0x7f42f0076750 2024-03-29T10:39:11.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.899+0000 7f43148f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:11.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.899+0000 7f43148f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:11.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.899+0000 7f43148f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:11.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.899+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f4005080 con 0x7f43100b17d0 2024-03-29T10:39:11.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.909+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4300050c90 con 0x7f43100b17d0 2024-03-29T10:39:11.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.909+0000 7f4306ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:11.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:11.909+0000 7f4306ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:12.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.225+0000 7f43148f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:12.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.226+0000 7f43148f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:12.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.226+0000 7f43148f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:12.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.226+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42f4005330 con 0x7f43100b17d0 2024-03-29T10:39:12.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.230+0000 7f4306ffd640 1 -- 172.21.15.117:0/332581725 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f4300050e30 con 0x7f43100b17d0 2024-03-29T10:39:12.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.230+0000 7f4306ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:12.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.230+0000 7f4306ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:12.215 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:39:12.215 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:12.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.233+0000 7f43148f6640 10 librados: watch_flush enter 2024-03-29T10:39:12.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.235+0000 7f43148f6640 10 librados: watch_flush exit 2024-03-29T10:39:12.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.235+0000 7f43148f6640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T10:39:12.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.235+0000 7f43148f6640 10 client.4384.objecter successfully canceled tick 2024-03-29T10:39:12.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.236+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 >> v1:172.21.15.117:6816/376768171 conn(0x7f42f0076750 legacy=0x7f42f0078c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:12.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.236+0000 7f43148f6640 10 monclient: shutdown 2024-03-29T10:39:12.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.236+0000 7f43148f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:12.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.236+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 >> v1:172.21.15.136:6789/0 conn(0x7f43100b17d0 legacy=0x7f43100af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:12.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.236+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 shutdown_connections 2024-03-29T10:39:12.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.236+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 >> 172.21.15.117:0/332581725 conn(0x7f4310066a00 msgr2=0x7f43100b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:12.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.236+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 shutdown_connections 2024-03-29T10:39:12.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.236+0000 7f43148f6640 1 -- 172.21.15.117:0/332581725 wait complete. 2024-03-29T10:39:12.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:12.237+0000 7f43148f6640 1 librados: shutdown 2024-03-29T10:39:12.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:19.047 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:19.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.219+0000 7f854e1c4640 10 monclient: get_monmap_and_config 2024-03-29T10:39:19.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.219+0000 7f854e1c4640 10 monclient: build_initial_monmap 2024-03-29T10:39:19.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.219+0000 7f854e1c4640 10 monclient: monmap: 2024-03-29T10:39:19.203 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:19.204 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:19.204 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:39:19.220085+0000 2024-03-29T10:39:19.204 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:39:19.220085+0000 2024-03-29T10:39:19.204 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:19.204 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:19.204 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:19.204 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:39:19.204 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:39:19.204 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:19.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.220+0000 7f854e1c4640 1 Processor -- start 2024-03-29T10:39:19.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.220+0000 7f854e1c4640 1 -- start start 2024-03-29T10:39:19.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.220+0000 7f854e1c4640 10 monclient: init 2024-03-29T10:39:19.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:19.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:19.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 10 monclient(hunting): picked mon.noname-c con 0x7f85480b4100 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:19.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 10 monclient(hunting): picked mon.noname-b con 0x7f854813ee90 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:19.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 10 monclient(hunting): picked mon.noname-a con 0x7f85480af900 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:19.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85480afd10 con 0x7f85480af900 2024-03-29T10:39:19.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85480af6f0 con 0x7f85480b4100 2024-03-29T10:39:19.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85480aac80 con 0x7f854813ee90 2024-03-29T10:39:19.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:19.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.221+0000 7f854e1c4640 10 monclient(hunting): authenticate will time out at 1080.483514s 2024-03-29T10:39:19.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.224+0000 7f8546ffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f854813ee90 0x7f854813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:42099/0 (socket says 172.21.15.117:42099) 2024-03-29T10:39:19.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.224+0000 7f8546ffd640 1 -- 172.21.15.117:0/3408245661 learned_addr learned my addr 172.21.15.117:0/3408245661 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:19.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736261572 0 0) 0x7f85480aac80 con 0x7f854813ee90 2024-03-29T10:39:19.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:19.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:19.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:19.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:39:19.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8530003520 con 0x7f854813ee90 2024-03-29T10:39:19.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507123034 0 0) 0x7f85480af6f0 con 0x7f85480b4100 2024-03-29T10:39:19.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.227+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85480aac80 con 0x7f85480b4100 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 666296088 0 0) 0x7f8530003520 con 0x7f854813ee90 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 >> v1:172.21.15.136:6789/0 conn(0x7f85480b4100 legacy=0x7f85480adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 >> v1:172.21.15.117:6789/0 conn(0x7f85480af900 legacy=0x7f854814be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 1 monclient: found mon.noname-b 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85480aaf60 con 0x7f854813ee90 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:19.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8540001a10 con 0x7f854813ee90 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f85467fc640 10 monclient: dump: 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:19.215 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f854e1c4640 5 monclient: authenticate success, global_id 4427 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.228+0000 7f854e1c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8540001b70 con 0x7f854813ee90 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f85467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f85467fc640 1 -- 172.21.15.117:0/3408245661 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8540001f80 con 0x7f854813ee90 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f85467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f85467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f85467fc640 10 monclient: dump: 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:19.216 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:19.217 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:19.217 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:19.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f854e1c4640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:19.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f854e1c4640 4 set_mon_vals no callback set 2024-03-29T10:39:19.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f854e1c4640 10 monclient: shutdown 2024-03-29T10:39:19.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f854e1c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:19.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.229+0000 7f854e1c4640 1 -- 172.21.15.117:0/3408245661 >> v1:172.21.15.117:6790/0 conn(0x7f854813ee90 legacy=0x7f854813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:19.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.233+0000 7f854e1c4640 1 -- 172.21.15.117:0/3408245661 shutdown_connections 2024-03-29T10:39:19.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.233+0000 7f854e1c4640 1 -- 172.21.15.117:0/3408245661 >> 172.21.15.117:0/3408245661 conn(0x7f85480b35e0 msgr2=0x7f85480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:19.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.233+0000 7f854e1c4640 1 -- 172.21.15.117:0/3408245661 shutdown_connections 2024-03-29T10:39:19.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.233+0000 7f854e1c4640 1 -- 172.21.15.117:0/3408245661 wait complete. 2024-03-29T10:39:19.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.234+0000 7f854e1c4640 10 monclient: build_initial_monmap 2024-03-29T10:39:19.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.234+0000 7f854e1c4640 10 monclient: monmap: 2024-03-29T10:39:19.219 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.234+0000 7f854e1c4640 1 librados: starting msgr at 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.235+0000 7f854e1c4640 1 librados: starting objecter 2024-03-29T10:39:19.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.235+0000 7f854e1c4640 1 Processor -- start 2024-03-29T10:39:19.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.236+0000 7f854e1c4640 1 -- start start 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.236+0000 7f854e1c4640 1 librados: setting wanted keys 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.236+0000 7f854e1c4640 1 librados: calling monclient init 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.236+0000 7f854e1c4640 10 monclient: init 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.236+0000 7f854e1c4640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.237+0000 7f854e1c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.237+0000 7f854e1c4640 10 monclient(hunting): picked mon.noname-a con 0x7f85480af900 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.237+0000 7f854e1c4640 10 monclient(hunting): picked mon.noname-b con 0x7f85480b4100 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.237+0000 7f854e1c4640 10 monclient(hunting): picked mon.noname-c con 0x7f854813ee90 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.237+0000 7f854e1c4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854814fb50 con 0x7f85480af900 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.237+0000 7f854e1c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854814fe30 con 0x7f85480b4100 2024-03-29T10:39:19.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.237+0000 7f854e1c4640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85481e4210 con 0x7f854813ee90 2024-03-29T10:39:19.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.237+0000 7f854e1c4640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:19.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.237+0000 7f854e1c4640 10 monclient(hunting): authenticate will time out at 1080.499178s 2024-03-29T10:39:19.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.244+0000 7f8547fff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f854813ee90 0x7f85481eec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:42203/0 (socket says 172.21.15.117:42203) 2024-03-29T10:39:19.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.244+0000 7f8547fff640 1 -- 172.21.15.117:0/1144422015 learned_addr learned my addr 172.21.15.117:0/1144422015 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:19.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.244+0000 7f852ffff640 10 client.?.objecter ms_handle_connect 0x7f854813ee90 2024-03-29T10:39:19.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.244+0000 7f852ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:19.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.246+0000 7f852ffff640 10 client.?.objecter ms_handle_connect 0x7f85480b4100 2024-03-29T10:39:19.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.246+0000 7f852ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:19.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.246+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978103712 0 0) 0x7f854814fe30 con 0x7f85480b4100 2024-03-29T10:39:19.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.246+0000 7f852ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:19.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.246+0000 7f852ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:19.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.246+0000 7f852ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:19.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.246+0000 7f852ffff640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:39:19.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.246+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534004eb0 con 0x7f85480b4100 2024-03-29T10:39:19.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.247+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677489010 0 0) 0x7f85481e4210 con 0x7f854813ee90 2024-03-29T10:39:19.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.247+0000 7f852ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:19.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.247+0000 7f852ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:19.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.247+0000 7f852ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:19.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.247+0000 7f852ffff640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:39:19.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.247+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f854814fe30 con 0x7f854813ee90 2024-03-29T10:39:19.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4160254820 0 0) 0x7f8534004eb0 con 0x7f85480b4100 2024-03-29T10:39:19.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85481e4210 con 0x7f85480b4100 2024-03-29T10:39:19.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 448870993 0 0) 0x7f854814fe30 con 0x7f854813ee90 2024-03-29T10:39:19.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8534004eb0 con 0x7f854813ee90 2024-03-29T10:39:19.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8540001f40 con 0x7f85480b4100 2024-03-29T10:39:19.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:19.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:19.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient(hunting): dump: 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8538003f40 con 0x7f854813ee90 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:19.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient(hunting): dump: 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 970745630 0 0) 0x7f8534004eb0 con 0x7f854813ee90 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 >> v1:172.21.15.136:6789/0 conn(0x7f85480b4100 legacy=0x7f8548071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 >> v1:172.21.15.117:6789/0 conn(0x7f85480af900 legacy=0x7f85481e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:19.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 monclient: found mon.c 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854806e7a0 con 0x7f854813ee90 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient: _finish_auth 0 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.248+0000 7f852ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f854e1c4640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f854e1c4640 10 monclient: _renew_subs 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f854e1c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85481e4e40 con 0x7f854813ee90 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f854e1c4640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f854e1c4640 10 monclient: _renew_subs 2024-03-29T10:39:19.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f854e1c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85480721f0 con 0x7f854813ee90 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f854e1c4640 1 librados: init done 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8538004450 con 0x7f854813ee90 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f852ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8538001070 con 0x7f854813ee90 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f852ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f852ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.251+0000 7f852ffff640 10 monclient: dump: 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:19.242 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:19.243 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:19.243 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:19.243 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:19.243 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:19.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.252+0000 7f8545ffb640 4 set_mon_vals no callback set 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.255+0000 7f854e1c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.255+0000 7f854e1c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.255+0000 7f854e1c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.255+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8528005080 con 0x7f854813ee90 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.257+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f85380011d0 con 0x7f854813ee90 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.258+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f85380848f0 con 0x7f854813ee90 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.258+0000 7f852ffff640 10 client.4430.objecter ms_dispatch 0x7f8548000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.258+0000 7f852ffff640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.258+0000 7f852ffff640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.258+0000 7f852ffff640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.259+0000 7f85477fe640 10 monclient: get_auth_request con 0x7f853406dea0 auth_method 0 2024-03-29T10:39:19.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.259+0000 7f852ffff640 10 client.4430.objecter ms_handle_connect 0x7f853406dea0 2024-03-29T10:39:19.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.270+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8538050000 con 0x7f854813ee90 2024-03-29T10:39:19.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.270+0000 7f852ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:19.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.270+0000 7f852ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:19.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.591+0000 7f854e1c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:19.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.591+0000 7f854e1c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:19.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.591+0000 7f854e1c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:19.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.592+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8528005330 con 0x7f854813ee90 2024-03-29T10:39:19.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.593+0000 7f852ffff640 1 -- 172.21.15.117:0/1144422015 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f8538052100 con 0x7f854813ee90 2024-03-29T10:39:19.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.593+0000 7f852ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:19.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.593+0000 7f852ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:19.577 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:39:19.577 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:19.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.596+0000 7f854e1c4640 10 librados: watch_flush enter 2024-03-29T10:39:19.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.596+0000 7f854e1c4640 10 librados: watch_flush exit 2024-03-29T10:39:19.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.596+0000 7f854e1c4640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T10:39:19.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.596+0000 7f854e1c4640 10 client.4430.objecter successfully canceled tick 2024-03-29T10:39:19.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.598+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 >> v1:172.21.15.117:6816/376768171 conn(0x7f853406dea0 legacy=0x7f8534070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:19.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.598+0000 7f854e1c4640 10 monclient: shutdown 2024-03-29T10:39:19.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.598+0000 7f854e1c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:19.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.598+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 >> v1:172.21.15.117:6790/0 conn(0x7f854813ee90 legacy=0x7f85481eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:19.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.600+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 shutdown_connections 2024-03-29T10:39:19.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.600+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 >> 172.21.15.117:0/1144422015 conn(0x7f85480b35e0 msgr2=0x7f85480b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:19.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.600+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 shutdown_connections 2024-03-29T10:39:19.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.600+0000 7f854e1c4640 1 -- 172.21.15.117:0/1144422015 wait complete. 2024-03-29T10:39:19.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:19.600+0000 7f854e1c4640 1 librados: shutdown 2024-03-29T10:39:19.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:26.412 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:26.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.581+0000 7f101754d640 10 monclient: get_monmap_and_config 2024-03-29T10:39:26.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.581+0000 7f101754d640 10 monclient: build_initial_monmap 2024-03-29T10:39:26.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.581+0000 7f101754d640 10 monclient: monmap: 2024-03-29T10:39:26.565 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:26.565 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:26.566 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:39:26.582344+0000 2024-03-29T10:39:26.566 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:39:26.582344+0000 2024-03-29T10:39:26.566 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:26.566 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:26.566 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:26.566 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:39:26.566 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:39:26.566 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:26.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.582+0000 7f101754d640 1 Processor -- start 2024-03-29T10:39:26.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.582+0000 7f101754d640 1 -- start start 2024-03-29T10:39:26.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.582+0000 7f101754d640 10 monclient: init 2024-03-29T10:39:26.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:26.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:26.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 10 monclient(hunting): picked mon.noname-a con 0x7f101014a400 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:26.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 10 monclient(hunting): picked mon.noname-c con 0x7f10100aa500 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:26.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 10 monclient(hunting): picked mon.noname-b con 0x7f1010149800 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:26.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10100aca50 con 0x7f101014a400 2024-03-29T10:39:26.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1010149cb0 con 0x7f10100aa500 2024-03-29T10:39:26.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10100acc80 con 0x7f1010149800 2024-03-29T10:39:26.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:26.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.583+0000 7f101754d640 10 monclient(hunting): authenticate will time out at 1087.846071s 2024-03-29T10:39:26.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.587+0000 7f1016d4c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f1010149800 0x7f1010151ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:59877/0 (socket says 172.21.15.117:59877) 2024-03-29T10:39:26.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.587+0000 7f1016d4c640 1 -- 172.21.15.117:0/1097510027 learned_addr learned my addr 172.21.15.117:0/1097510027 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:26.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985952161 0 0) 0x7f10100acc80 con 0x7f1010149800 2024-03-29T10:39:26.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient(hunting): _negotiate 2024-03-29T10:39:26.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:26.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:26.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:39:26.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ffc003520 con 0x7f1010149800 2024-03-29T10:39:26.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734025828 0 0) 0x7f10100aca50 con 0x7f101014a400 2024-03-29T10:39:26.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient(hunting): _negotiate 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10100acc80 con 0x7f101014a400 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3376632212 0 0) 0x7f0ffc003520 con 0x7f1010149800 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 >> v1:172.21.15.136:6789/0 conn(0x7f10100aa500 legacy=0x7f10100ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 >> v1:172.21.15.117:6789/0 conn(0x7f101014a400 legacy=0x7f10100aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient: _finish_hunting 0 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 monclient: found mon.noname-b 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10100ac350 con 0x7f1010149800 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient: _finish_auth 0 2024-03-29T10:39:26.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient: _check_auth_tickets 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f100c001610 con 0x7f1010149800 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f1015549640 10 monclient: dump: 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:26.579 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f101754d640 5 monclient: authenticate success, global_id 4433 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.591+0000 7f101754d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.592+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f100c001770 con 0x7f1010149800 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.592+0000 7f1015549640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.592+0000 7f1015549640 1 -- 172.21.15.117:0/1097510027 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f100c001b80 con 0x7f1010149800 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.592+0000 7f101754d640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.592+0000 7f101754d640 4 set_mon_vals no callback set 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.592+0000 7f1015549640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.592+0000 7f1015549640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.592+0000 7f1015549640 10 monclient: dump: 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.580 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.593+0000 7f101754d640 10 monclient: shutdown 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.593+0000 7f101754d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.593+0000 7f101754d640 1 -- 172.21.15.117:0/1097510027 >> v1:172.21.15.117:6790/0 conn(0x7f1010149800 legacy=0x7f1010151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.593+0000 7f101754d640 1 -- 172.21.15.117:0/1097510027 shutdown_connections 2024-03-29T10:39:26.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.593+0000 7f101754d640 1 -- 172.21.15.117:0/1097510027 >> 172.21.15.117:0/1097510027 conn(0x7f10100b3a30 msgr2=0x7f10100b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:26.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.595+0000 7f101754d640 1 -- 172.21.15.117:0/1097510027 shutdown_connections 2024-03-29T10:39:26.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.596+0000 7f101754d640 1 -- 172.21.15.117:0/1097510027 wait complete. 2024-03-29T10:39:26.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.597+0000 7f101754d640 10 monclient: build_initial_monmap 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.597+0000 7f101754d640 10 monclient: monmap: 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.597+0000 7f101754d640 1 librados: starting msgr at 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.597+0000 7f101754d640 1 librados: starting objecter 2024-03-29T10:39:26.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.597+0000 7f101754d640 1 Processor -- start 2024-03-29T10:39:26.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.600+0000 7f101754d640 1 -- start start 2024-03-29T10:39:26.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.600+0000 7f101754d640 1 librados: setting wanted keys 2024-03-29T10:39:26.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.600+0000 7f101754d640 1 librados: calling monclient init 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.600+0000 7f101754d640 10 monclient: init 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 10 monclient(hunting): picked mon.noname-c con 0x7f10100aa500 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 10 monclient(hunting): picked mon.noname-b con 0x7f1010149800 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 10 monclient(hunting): picked mon.noname-a con 0x7f101014a400 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1010145f40 con 0x7f101014a400 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1010147000 con 0x7f1010149800 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1010147270 con 0x7f10100aa500 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:26.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.601+0000 7f101754d640 10 monclient(hunting): authenticate will time out at 1087.863962s 2024-03-29T10:39:26.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.606+0000 7f1016d4c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f101014a400 0x7f10101e5af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:55591/0 (socket says 172.21.15.117:55591) 2024-03-29T10:39:26.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.606+0000 7f1016d4c640 1 -- 172.21.15.117:0/2257887455 learned_addr learned my addr 172.21.15.117:0/2257887455 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:26.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.606+0000 7f1006ffd640 10 client.?.objecter ms_handle_connect 0x7f101014a400 2024-03-29T10:39:26.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.606+0000 7f1006ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:26.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.608+0000 7f1006ffd640 10 client.?.objecter ms_handle_connect 0x7f10100aa500 2024-03-29T10:39:26.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.608+0000 7f1006ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:26.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.608+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984051689 0 0) 0x7f1010147270 con 0x7f10100aa500 2024-03-29T10:39:26.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.608+0000 7f1006ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:26.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.608+0000 7f1006ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:26.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.608+0000 7f1006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:26.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.608+0000 7f1006ffd640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:39:26.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.608+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff4004eb0 con 0x7f10100aa500 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074298240 0 0) 0x7f1010145f40 con 0x7f101014a400 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1010147270 con 0x7f101014a400 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2466205395 0 0) 0x7f0ff4004eb0 con 0x7f10100aa500 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1010145f40 con 0x7f10100aa500 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1000001850 con 0x7f10100aa500 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:26.596 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2798252631 0 0) 0x7f1010147270 con 0x7f101014a400 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ff4004eb0 con 0x7f101014a400 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f100c001a80 con 0x7f101014a400 2024-03-29T10:39:26.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:26.598 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 959196218 0 0) 0x7f0ff4004eb0 con 0x7f101014a400 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 >> v1:172.21.15.117:6790/0 conn(0x7f10100aa500 legacy=0x7f10101450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.609+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 >> v1:172.21.15.136:6789/0 conn(0x7f1010149800 legacy=0x7f10101387d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f1006ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f1006ffd640 1 monclient: found mon.a 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f1006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f1006ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10101463d0 con 0x7f101014a400 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f1006ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f1006ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f1006ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f101754d640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:39:26.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f101754d640 10 monclient: _renew_subs 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f101754d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.610+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1010145900 con 0x7f101014a400 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f101754d640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f101754d640 10 monclient: _renew_subs 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f101754d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1010138ee0 con 0x7f101014a400 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f101754d640 1 librados: init done 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f100c004a90 con 0x7f101014a400 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f1006ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f100c004da0 con 0x7f101014a400 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f1006ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f1006ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.611+0000 7f1006ffd640 10 monclient: dump: 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:26.600 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:26.601 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.601 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:26.601 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.601 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:26.601 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:26.601 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:26.601 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:26.601 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:26.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.612+0000 7f101754d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:26.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.612+0000 7f101754d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:26.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.612+0000 7f101754d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:39:26.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.612+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101004f730 con 0x7f101014a400 2024-03-29T10:39:26.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.613+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f100c01ad70 con 0x7f101014a400 2024-03-29T10:39:26.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.613+0000 7f1014d48640 4 set_mon_vals no callback set 2024-03-29T10:39:26.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.613+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f100c084b00 con 0x7f101014a400 2024-03-29T10:39:26.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.613+0000 7f1006ffd640 10 client.4413.objecter ms_dispatch 0x7f1010000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:26.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.613+0000 7f1006ffd640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:26.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.613+0000 7f1006ffd640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:26.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.613+0000 7f101654b640 10 monclient: get_auth_request con 0x7f0ff406de80 auth_method 0 2024-03-29T10:39:26.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.613+0000 7f1006ffd640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T10:39:26.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.613+0000 7f1006ffd640 10 client.4413.objecter ms_handle_connect 0x7f0ff406de80 2024-03-29T10:39:26.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.622+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f100c0503e0 con 0x7f101014a400 2024-03-29T10:39:26.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.622+0000 7f1006ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:26.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.622+0000 7f1006ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:26.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.938+0000 7f101754d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:26.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.938+0000 7f101754d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:26.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.938+0000 7f101754d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:39:26.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.938+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10101391e0 con 0x7f101014a400 2024-03-29T10:39:26.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.939+0000 7f1006ffd640 1 -- 172.21.15.117:0/2257887455 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f100c052310 con 0x7f101014a400 2024-03-29T10:39:26.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.939+0000 7f1006ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:26.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.939+0000 7f1006ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:26.923 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:39:26.924 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:26.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.943+0000 7f101754d640 10 librados: watch_flush enter 2024-03-29T10:39:26.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.943+0000 7f101754d640 10 librados: watch_flush exit 2024-03-29T10:39:26.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.943+0000 7f101754d640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T10:39:26.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.943+0000 7f101754d640 10 client.4413.objecter successfully canceled tick 2024-03-29T10:39:26.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.943+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 >> v1:172.21.15.117:6816/376768171 conn(0x7f0ff406de80 legacy=0x7f0ff4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.943+0000 7f101754d640 10 monclient: shutdown 2024-03-29T10:39:26.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.943+0000 7f101754d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:26.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.943+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 >> v1:172.21.15.117:6789/0 conn(0x7f101014a400 legacy=0x7f10101e5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.945+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 shutdown_connections 2024-03-29T10:39:26.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.945+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 >> 172.21.15.117:0/2257887455 conn(0x7f10100b3a30 msgr2=0x7f10100b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:26.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.945+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 shutdown_connections 2024-03-29T10:39:26.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.945+0000 7f101754d640 1 -- 172.21.15.117:0/2257887455 wait complete. 2024-03-29T10:39:26.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:26.946+0000 7f101754d640 1 librados: shutdown 2024-03-29T10:39:26.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:33.754 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:33.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.920+0000 7fbae2a24640 10 monclient: get_monmap_and_config 2024-03-29T10:39:33.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.920+0000 7fbae2a24640 10 monclient: build_initial_monmap 2024-03-29T10:39:33.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.920+0000 7fbae2a24640 10 monclient: monmap: 2024-03-29T10:39:33.904 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:33.904 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:39:33.921200+0000 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:39:33.921200+0000 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.921+0000 7fbae2a24640 1 Processor -- start 2024-03-29T10:39:33.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.921+0000 7fbae2a24640 1 -- start start 2024-03-29T10:39:33.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.921+0000 7fbae2a24640 10 monclient: init 2024-03-29T10:39:33.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:33.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:33.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 10 monclient(hunting): picked mon.noname-c con 0x7fbadc0aad20 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:33.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 10 monclient(hunting): picked mon.noname-b con 0x7fbadc06d8c0 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:33.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 10 monclient(hunting): picked mon.noname-a con 0x7fbadc0afc40 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:33.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc0af4d0 con 0x7fbadc0afc40 2024-03-29T10:39:33.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc0af7b0 con 0x7fbadc0aad20 2024-03-29T10:39:33.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc0acab0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:33.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.922+0000 7fbae2a24640 10 monclient(hunting): authenticate will time out at 1095.184799s 2024-03-29T10:39:33.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.925+0000 7fbae1a22640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fbadc0aad20 0x7fbadc0ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:46692/0 (socket says 172.21.15.117:46692) 2024-03-29T10:39:33.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.925+0000 7fbae1a22640 1 -- 172.21.15.117:0/147135069 learned_addr learned my addr 172.21.15.117:0/147135069 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802391090 0 0) 0x7fbadc0af7b0 con 0x7fbadc0aad20 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 10 monclient(hunting): _negotiate 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac8003520 con 0x7fbadc0aad20 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356290090 0 0) 0x7fbadc0acab0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 10 monclient(hunting): _negotiate 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:33.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:33.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:39:33.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbadc0af7b0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3455116282 0 0) 0x7fbac8003520 con 0x7fbadc0aad20 2024-03-29T10:39:33.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 >> v1:172.21.15.117:6790/0 conn(0x7fbadc06d8c0 legacy=0x7fbadc0a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:33.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.926+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 >> v1:172.21.15.117:6789/0 conn(0x7fbadc0afc40 legacy=0x7fbadc141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:33.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 10 monclient: _finish_hunting 0 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 1 monclient: found mon.noname-c 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc0aba30 con 0x7fbadc0aad20 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 10 monclient: _finish_auth 0 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 10 monclient: _check_auth_tickets 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae2a24640 5 monclient: authenticate success, global_id 4393 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae2a24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbacc0015c0 con 0x7fbadc0aad20 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae0a20640 10 monclient: dump: 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:33.912 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:33.913 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.913 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.913 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.913 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:33.913 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:33.913 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:33.913 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:33.913 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:33.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.927+0000 7fbae2a24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.929+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbacc001720 con 0x7fbadc0aad20 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.929+0000 7fbae0a20640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.929+0000 7fbae2a24640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.929+0000 7fbae2a24640 4 set_mon_vals no callback set 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.930+0000 7fbae0a20640 1 -- 172.21.15.117:0/147135069 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbacc001ab0 con 0x7fbadc0aad20 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.930+0000 7fbae0a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.930+0000 7fbae0a20640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.930+0000 7fbae0a20640 10 monclient: dump: 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:33.914 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.930+0000 7fbae2a24640 10 monclient: shutdown 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.930+0000 7fbae2a24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:33.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.930+0000 7fbae2a24640 1 -- 172.21.15.117:0/147135069 >> v1:172.21.15.136:6789/0 conn(0x7fbadc0aad20 legacy=0x7fbadc0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.930+0000 7fbae2a24640 1 -- 172.21.15.117:0/147135069 shutdown_connections 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.930+0000 7fbae2a24640 1 -- 172.21.15.117:0/147135069 >> 172.21.15.117:0/147135069 conn(0x7fbadc0673e0 msgr2=0x7fbadc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.931+0000 7fbae2a24640 1 -- 172.21.15.117:0/147135069 shutdown_connections 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.931+0000 7fbae2a24640 1 -- 172.21.15.117:0/147135069 wait complete. 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.932+0000 7fbae2a24640 10 monclient: build_initial_monmap 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.932+0000 7fbae2a24640 10 monclient: monmap: 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:39:33.916 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:39:33.917 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:33.917 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:33.917 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:33.917 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:39:33.917 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:39:33.917 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:33.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.932+0000 7fbae2a24640 1 librados: starting msgr at 2024-03-29T10:39:33.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.932+0000 7fbae2a24640 1 librados: starting objecter 2024-03-29T10:39:33.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.933+0000 7fbae2a24640 1 Processor -- start 2024-03-29T10:39:33.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.933+0000 7fbae2a24640 1 -- start start 2024-03-29T10:39:33.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.933+0000 7fbae2a24640 1 librados: setting wanted keys 2024-03-29T10:39:33.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.933+0000 7fbae2a24640 1 librados: calling monclient init 2024-03-29T10:39:33.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.933+0000 7fbae2a24640 10 monclient: init 2024-03-29T10:39:33.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.933+0000 7fbae2a24640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:33.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.933+0000 7fbae2a24640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:33.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.933+0000 7fbae2a24640 10 monclient(hunting): picked mon.noname-c con 0x7fbadc06d8c0 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:33.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.933+0000 7fbae2a24640 10 monclient(hunting): picked mon.noname-b con 0x7fbadc0aad20 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:33.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.934+0000 7fbae2a24640 10 monclient(hunting): picked mon.noname-a con 0x7fbadc0afc40 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:33.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.934+0000 7fbae2a24640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc1e6ec0 con 0x7fbadc0afc40 2024-03-29T10:39:33.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.934+0000 7fbae2a24640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc1e71a0 con 0x7fbadc0aad20 2024-03-29T10:39:33.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.934+0000 7fbae2a24640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc142da0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.934+0000 7fbae2a24640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:33.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.934+0000 7fbae2a24640 10 monclient(hunting): authenticate will time out at 1095.196137s 2024-03-29T10:39:33.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.934+0000 7fbae1221640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fbadc0aad20 0x7fbadc1e7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:47784/0 (socket says 172.21.15.117:47784) 2024-03-29T10:39:33.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.934+0000 7fbae1221640 1 -- 172.21.15.117:0/1672540973 learned_addr learned my addr 172.21.15.117:0/1672540973 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:33.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 10 client.?.objecter ms_handle_connect 0x7fbadc0aad20 2024-03-29T10:39:33.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:33.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 10 client.?.objecter ms_handle_connect 0x7fbadc06d8c0 2024-03-29T10:39:33.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854969983 0 0) 0x7fbadc1e71a0 con 0x7fbadc0aad20 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.937+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac0004eb0 con 0x7fbadc0aad20 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260575471 0 0) 0x7fbadc142da0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:39:33.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbadc1e71a0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 930478578 0 0) 0x7fbac0004eb0 con 0x7fbadc0aad20 2024-03-29T10:39:33.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbadc142da0 con 0x7fbadc0aad20 2024-03-29T10:39:33.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbad4001cc0 con 0x7fbadc0aad20 2024-03-29T10:39:33.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:33.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): dump: 2024-03-29T10:39:33.923 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:33.923 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1506133648 0 0) 0x7fbadc1e71a0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbac0004eb0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbacc001a30 con 0x7fbadc06d8c0 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:33.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient(hunting): dump: 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3145314009 0 0) 0x7fbac0004eb0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 >> v1:172.21.15.136:6789/0 conn(0x7fbadc0aad20 legacy=0x7fbadc1e7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 >> v1:172.21.15.117:6789/0 conn(0x7fbadc0afc40 legacy=0x7fbadc13e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:33.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 monclient: found mon.c 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.938+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc1e7d40 con 0x7fbadc06d8c0 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.939+0000 7fbad27fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.939+0000 7fbad27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.939+0000 7fbad27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.939+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbacc004a90 con 0x7fbadc06d8c0 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.939+0000 7fbad27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.939+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbacc004da0 con 0x7fbadc06d8c0 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.939+0000 7fbad27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.939+0000 7fbad27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.939+0000 7fbad27fc640 10 monclient: dump: 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:33.926 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:33.927 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.927 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:33.927 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.927 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:33.927 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:33.927 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:33.927 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:33.927 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:33.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.940+0000 7fbae2a24640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:39:33.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.940+0000 7fbae2a24640 10 monclient: _renew_subs 2024-03-29T10:39:33.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.940+0000 7fbae2a24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:33.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.940+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbadc06dd10 con 0x7fbadc06d8c0 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.940+0000 7fbae2a24640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.940+0000 7fbae2a24640 10 monclient: _renew_subs 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.940+0000 7fbae2a24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.940+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbadc070a10 con 0x7fbadc06d8c0 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.940+0000 7fbae2a24640 1 librados: init done 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.941+0000 7fbad37fe640 4 set_mon_vals no callback set 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.942+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fbacc01ad70 con 0x7fbadc06d8c0 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.942+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fbacc084a60 con 0x7fbadc06d8c0 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.942+0000 7fbad27fc640 10 client.4442.objecter ms_dispatch 0x7fbadc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.942+0000 7fbad27fc640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.942+0000 7fbad27fc640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.943+0000 7fbad27fc640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.943+0000 7fbae1221640 10 monclient: get_auth_request con 0x7fbac006ddc0 auth_method 0 2024-03-29T10:39:33.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.943+0000 7fbad27fc640 10 client.4442.objecter ms_handle_connect 0x7fbac006ddc0 2024-03-29T10:39:33.931 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.943+0000 7fbae2a24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:33.931 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.943+0000 7fbae2a24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:33.931 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.943+0000 7fbae2a24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:33.931 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.943+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac4005080 con 0x7fbadc06d8c0 2024-03-29T10:39:33.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.952+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbacc050340 con 0x7fbadc06d8c0 2024-03-29T10:39:33.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.952+0000 7fbad27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:33.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:33.952+0000 7fbad27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:34.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.290+0000 7fbae2a24640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:34.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.290+0000 7fbae2a24640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:34.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.290+0000 7fbae2a24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:34.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.290+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbac4005330 con 0x7fbadc06d8c0 2024-03-29T10:39:34.279 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:39:34.279 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:34.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.291+0000 7fbad27fc640 1 -- 172.21.15.117:0/1672540973 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fbacc001140 con 0x7fbadc06d8c0 2024-03-29T10:39:34.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.292+0000 7fbad27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:34.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.292+0000 7fbad27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:34.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.295+0000 7fbae2a24640 10 librados: watch_flush enter 2024-03-29T10:39:34.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.296+0000 7fbae2a24640 10 librados: watch_flush exit 2024-03-29T10:39:34.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.296+0000 7fbae2a24640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-29T10:39:34.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.296+0000 7fbae2a24640 10 client.4442.objecter successfully canceled tick 2024-03-29T10:39:34.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.296+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 >> v1:172.21.15.117:6816/376768171 conn(0x7fbac006ddc0 legacy=0x7fbac0070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:34.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.296+0000 7fbae2a24640 10 monclient: shutdown 2024-03-29T10:39:34.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.296+0000 7fbae2a24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:34.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.296+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 >> v1:172.21.15.117:6790/0 conn(0x7fbadc06d8c0 legacy=0x7fbadc0713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:34.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.297+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 shutdown_connections 2024-03-29T10:39:34.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.297+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 >> 172.21.15.117:0/1672540973 conn(0x7fbadc0673e0 msgr2=0x7fbadc0ac3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:34.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.297+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 shutdown_connections 2024-03-29T10:39:34.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.297+0000 7fbae2a24640 1 -- 172.21.15.117:0/1672540973 wait complete. 2024-03-29T10:39:34.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:34.297+0000 7fbae2a24640 1 librados: shutdown 2024-03-29T10:39:34.300 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:41.107 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:41.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.251+0000 7f861dc01640 10 monclient: get_monmap_and_config 2024-03-29T10:39:41.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.251+0000 7f861dc01640 10 monclient: build_initial_monmap 2024-03-29T10:39:41.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.251+0000 7f861dc01640 10 monclient: monmap: 2024-03-29T10:39:41.235 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:41.236 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:41.236 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:39:41.252192+0000 2024-03-29T10:39:41.236 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:39:41.252192+0000 2024-03-29T10:39:41.236 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:41.236 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:41.236 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:41.236 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:39:41.236 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:39:41.236 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 1 Processor -- start 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 1 -- start start 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 10 monclient: init 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 10 monclient(hunting): picked mon.noname-b con 0x7f86181457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 10 monclient(hunting): picked mon.noname-c con 0x7f861813f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 10 monclient(hunting): picked mon.noname-a con 0x7f86180a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86180aaef0 con 0x7f86180a9d70 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8618145bd0 con 0x7f861813f650 2024-03-29T10:39:41.238 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8618145eb0 con 0x7f86181457f0 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.252+0000 7f861dc01640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f861dc01640 10 monclient(hunting): authenticate will time out at 1102.515097s 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86177fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f86181457f0 0x7f861813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34784/0 (socket says 172.21.15.117:34784) 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86177fe640 1 -- 172.21.15.117:0/1597707442 learned_addr learned my addr 172.21.15.117:0/1597707442 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919882448 0 0) 0x7f86180aaef0 con 0x7f86180a9d70 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8600003520 con 0x7f86180a9d70 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152942648 0 0) 0x7f8618145eb0 con 0x7f86181457f0 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:41.239 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:41.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:41.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:39:41.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86180aaef0 con 0x7f86181457f0 2024-03-29T10:39:41.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2534256046 0 0) 0x7f8600003520 con 0x7f86180a9d70 2024-03-29T10:39:41.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 >> v1:172.21.15.117:6790/0 conn(0x7f86181457f0 legacy=0x7f861813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:41.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.253+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 >> v1:172.21.15.136:6789/0 conn(0x7f861813f650 legacy=0x7f86180b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:41.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:41.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 1 monclient: found mon.noname-a 2024-03-29T10:39:41.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8618146340 con 0x7f86180a9d70 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8610001a10 con 0x7f86180a9d70 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: dump: 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.242 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f861dc01640 5 monclient: authenticate success, global_id 4416 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f861dc01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8610001b70 con 0x7f86180a9d70 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 1 -- 172.21.15.117:0/1597707442 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8610001f80 con 0x7f86180a9d70 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f86167fc640 10 monclient: dump: 2024-03-29T10:39:41.243 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f861dc01640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.256+0000 7f861dc01640 4 set_mon_vals no callback set 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.257+0000 7f861dc01640 10 monclient: shutdown 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.257+0000 7f861dc01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:41.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.257+0000 7f861dc01640 1 -- 172.21.15.117:0/1597707442 >> v1:172.21.15.117:6789/0 conn(0x7f86180a9d70 legacy=0x7f861814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:41.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.257+0000 7f861dc01640 1 -- 172.21.15.117:0/1597707442 shutdown_connections 2024-03-29T10:39:41.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.257+0000 7f861dc01640 1 -- 172.21.15.117:0/1597707442 >> 172.21.15.117:0/1597707442 conn(0x7f8618066a00 msgr2=0x7f8618066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:41.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.258+0000 7f861dc01640 1 -- 172.21.15.117:0/1597707442 shutdown_connections 2024-03-29T10:39:41.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.258+0000 7f861dc01640 1 -- 172.21.15.117:0/1597707442 wait complete. 2024-03-29T10:39:41.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.259+0000 7f861dc01640 10 monclient: build_initial_monmap 2024-03-29T10:39:41.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.259+0000 7f861dc01640 10 monclient: monmap: 2024-03-29T10:39:41.251 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:41.251 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:41.251 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:39:41.251 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:39:41.251 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:41.251 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.259+0000 7f861dc01640 1 librados: starting msgr at 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.259+0000 7f861dc01640 1 librados: starting objecter 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.260+0000 7f861dc01640 1 Processor -- start 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.260+0000 7f861dc01640 1 -- start start 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.260+0000 7f861dc01640 1 librados: setting wanted keys 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.260+0000 7f861dc01640 1 librados: calling monclient init 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.260+0000 7f861dc01640 10 monclient: init 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f861dc01640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f861dc01640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:41.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f861dc01640 10 monclient(hunting): picked mon.noname-a con 0x7f86180a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f861dc01640 10 monclient(hunting): picked mon.noname-b con 0x7f861813f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f861dc01640 10 monclient(hunting): picked mon.noname-c con 0x7f86181457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f86177fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f86180a9d70 0x7f86181db710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:53272/0 (socket says 172.21.15.117:53272) 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f8617fff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f86181457f0 0x7f86181370f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:35302/0 (socket says 172.21.15.117:35302) 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f86177fe640 1 -- 172.21.15.117:0/2092941267 learned_addr learned my addr 172.21.15.117:0/2092941267 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86181db1f0 con 0x7f86180a9d70 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86180ab600 con 0x7f861813f650 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.261+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86180ab790 con 0x7f86181457f0 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f861dc01640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f861dc01640 10 monclient(hunting): authenticate will time out at 1102.524104s 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 client.?.objecter ms_handle_connect 0x7f86180a9d70 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 client.?.objecter ms_handle_connect 0x7f86181457f0 2024-03-29T10:39:41.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147917303 0 0) 0x7f86181db1f0 con 0x7f86180a9d70 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8604004eb0 con 0x7f86180a9d70 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499483759 0 0) 0x7f86180ab790 con 0x7f86181457f0 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86181db1f0 con 0x7f86181457f0 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1147695832 0 0) 0x7f86181db1f0 con 0x7f86181457f0 2024-03-29T10:39:41.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86180ab790 con 0x7f86181457f0 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8610001e80 con 0x7f86181457f0 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): dump: 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:41.255 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3442199878 0 0) 0x7f8604004eb0 con 0x7f86180a9d70 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86181db1f0 con 0x7f86180a9d70 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8608001890 con 0x7f86180a9d70 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.262+0000 7f85fffff640 10 monclient(hunting): dump: 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.256 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.263+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2592737614 0 0) 0x7f86180ab790 con 0x7f86181457f0 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.263+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 >> v1:172.21.15.136:6789/0 conn(0x7f861813f650 legacy=0x7f86181d7860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 >> v1:172.21.15.117:6789/0 conn(0x7f86180a9d70 legacy=0x7f86181db710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 1 monclient: found mon.c 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:41.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86181dad60 con 0x7f86181457f0 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 10 monclient: _finish_auth 0 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f861dc01640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f861dc01640 10 monclient: _renew_subs 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f861dc01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86181dafb0 con 0x7f86181457f0 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8610003c20 con 0x7f86181457f0 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8610003e10 con 0x7f86181457f0 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f861dc01640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:41.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f85fffff640 10 monclient: dump: 2024-03-29T10:39:41.259 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:41.259 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:41.259 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.259 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:41.259 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:41.259 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:41.259 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:41.259 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:41.259 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f8615ffb640 4 set_mon_vals no callback set 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f861dc01640 10 monclient: _renew_subs 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f861dc01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86180abb80 con 0x7f86181457f0 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.264+0000 7f861dc01640 1 librados: init done 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.266+0000 7f861dc01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.266+0000 7f861dc01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.266+0000 7f861dc01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:41.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.266+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8618075f10 con 0x7f86181457f0 2024-03-29T10:39:41.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.267+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f8610003f70 con 0x7f86181457f0 2024-03-29T10:39:41.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.268+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f8610084c70 con 0x7f86181457f0 2024-03-29T10:39:41.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.268+0000 7f85fffff640 10 client.4448.objecter ms_dispatch 0x7f8618000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:41.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.268+0000 7f85fffff640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:41.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.268+0000 7f85fffff640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:41.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.268+0000 7f85fffff640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T10:39:41.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.268+0000 7f86177fe640 10 monclient: get_auth_request con 0x7f860406dda0 auth_method 0 2024-03-29T10:39:41.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.269+0000 7f85fffff640 10 client.4448.objecter ms_handle_connect 0x7f860406dda0 2024-03-29T10:39:41.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.275+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8610050550 con 0x7f86181457f0 2024-03-29T10:39:41.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.275+0000 7f85fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:41.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.275+0000 7f85fffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:41.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.560+0000 7f861dc01640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:41.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.560+0000 7f861dc01640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:41.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.560+0000 7f861dc01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:41.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.560+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86181db380 con 0x7f86181457f0 2024-03-29T10:39:41.545 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:39:41.545 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:41.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.561+0000 7f85fffff640 1 -- 172.21.15.117:0/2092941267 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f8610052480 con 0x7f86181457f0 2024-03-29T10:39:41.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.561+0000 7f85fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:41.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.561+0000 7f85fffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:41.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.564+0000 7f861dc01640 10 librados: watch_flush enter 2024-03-29T10:39:41.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 10 librados: watch_flush exit 2024-03-29T10:39:41.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T10:39:41.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 10 client.4448.objecter successfully canceled tick 2024-03-29T10:39:41.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 >> v1:172.21.15.117:6816/376768171 conn(0x7f860406dda0 legacy=0x7f8604070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:41.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 10 monclient: shutdown 2024-03-29T10:39:41.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:41.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 >> v1:172.21.15.117:6790/0 conn(0x7f86181457f0 legacy=0x7f86181370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:41.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 shutdown_connections 2024-03-29T10:39:41.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 >> 172.21.15.117:0/2092941267 conn(0x7f8618066a00 msgr2=0x7f86180b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:41.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 shutdown_connections 2024-03-29T10:39:41.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 1 -- 172.21.15.117:0/2092941267 wait complete. 2024-03-29T10:39:41.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:41.565+0000 7f861dc01640 1 librados: shutdown 2024-03-29T10:39:41.565 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:48.372 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:48.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.532+0000 7f92fc5e6640 10 monclient: get_monmap_and_config 2024-03-29T10:39:48.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.532+0000 7f92fc5e6640 10 monclient: build_initial_monmap 2024-03-29T10:39:48.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.533+0000 7f92fc5e6640 10 monclient: monmap: 2024-03-29T10:39:48.517 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:48.517 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:48.517 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:39:48.533929+0000 2024-03-29T10:39:48.517 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:39:48.533929+0000 2024-03-29T10:39:48.517 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:48.517 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:48.518 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:48.518 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:39:48.518 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:39:48.518 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:48.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.534+0000 7f92fc5e6640 1 Processor -- start 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.534+0000 7f92fc5e6640 1 -- start start 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.534+0000 7f92fc5e6640 10 monclient: init 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 10 monclient(hunting): picked mon.noname-c con 0x7f92f41457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 10 monclient(hunting): picked mon.noname-b con 0x7f92f413f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 10 monclient(hunting): picked mon.noname-a con 0x7f92f40a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f40aaef0 con 0x7f92f40a9d70 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f4145bd0 con 0x7f92f41457f0 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f4145eb0 con 0x7f92f413f650 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fc5e6640 10 monclient(hunting): authenticate will time out at 1109.797269s 2024-03-29T10:39:48.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fab83640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f92f40a9d70 0x7f92f414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:53056/0 (socket says 172.21.15.117:53056) 2024-03-29T10:39:48.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.535+0000 7f92fab83640 1 -- 172.21.15.117:0/794377057 learned_addr learned my addr 172.21.15.117:0/794377057 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:48.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519159982 0 0) 0x7f92f40aaef0 con 0x7f92f40a9d70 2024-03-29T10:39:48.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 10 monclient(hunting): _negotiate 2024-03-29T10:39:48.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:48.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:48.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:39:48.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e0003520 con 0x7f92f40a9d70 2024-03-29T10:39:48.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362836960 0 0) 0x7f92f4145bd0 con 0x7f92f41457f0 2024-03-29T10:39:48.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 10 monclient(hunting): _negotiate 2024-03-29T10:39:48.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:48.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:48.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f40aaef0 con 0x7f92f41457f0 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4274254820 0 0) 0x7f92e0003520 con 0x7f92f40a9d70 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 >> v1:172.21.15.117:6790/0 conn(0x7f92f413f650 legacy=0x7f92f40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.536+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 >> v1:172.21.15.136:6789/0 conn(0x7f92f41457f0 legacy=0x7f92f413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: _finish_hunting 0 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 1 monclient: found mon.noname-a 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f4146340 con 0x7f92f40a9d70 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: _finish_auth 0 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: _check_auth_tickets 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:48.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f92ec001a10 con 0x7f92f40a9d70 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92fc5e6640 5 monclient: authenticate success, global_id 4422 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92fc5e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: dump: 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.533 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92fc5e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92ec001c30 con 0x7f92f40a9d70 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 1 -- 172.21.15.117:0/794377057 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f92ec001fc0 con 0x7f92f40a9d70 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92f9380640 10 monclient: dump: 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:48.534 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92fc5e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:48.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.537+0000 7f92fc5e6640 4 set_mon_vals no callback set 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.538+0000 7f92fc5e6640 10 monclient: shutdown 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.538+0000 7f92fc5e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.538+0000 7f92fc5e6640 1 -- 172.21.15.117:0/794377057 >> v1:172.21.15.117:6789/0 conn(0x7f92f40a9d70 legacy=0x7f92f414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.538+0000 7f92fc5e6640 1 -- 172.21.15.117:0/794377057 shutdown_connections 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.538+0000 7f92fc5e6640 1 -- 172.21.15.117:0/794377057 >> 172.21.15.117:0/794377057 conn(0x7f92f4066a00 msgr2=0x7f92f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.538+0000 7f92fc5e6640 1 -- 172.21.15.117:0/794377057 shutdown_connections 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.538+0000 7f92fc5e6640 1 -- 172.21.15.117:0/794377057 wait complete. 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.539+0000 7f92fc5e6640 10 monclient: build_initial_monmap 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.539+0000 7f92fc5e6640 10 monclient: monmap: 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:39:48.536 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.540+0000 7f92fc5e6640 1 librados: starting msgr at 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.540+0000 7f92fc5e6640 1 librados: starting objecter 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.540+0000 7f92fc5e6640 1 Processor -- start 2024-03-29T10:39:48.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.540+0000 7f92fc5e6640 1 -- start start 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.540+0000 7f92fc5e6640 1 librados: setting wanted keys 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.540+0000 7f92fc5e6640 1 librados: calling monclient init 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.540+0000 7f92fc5e6640 10 monclient: init 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 10 monclient(hunting): picked mon.noname-b con 0x7f92f40a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 10 monclient(hunting): picked mon.noname-a con 0x7f92f413f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 10 monclient(hunting): picked mon.noname-c con 0x7f92f41457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f41452b0 con 0x7f92f413f650 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f4140ac0 con 0x7f92f40a9d70 2024-03-29T10:39:48.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f4143320 con 0x7f92f41457f0 2024-03-29T10:39:48.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:48.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fc5e6640 10 monclient(hunting): authenticate will time out at 1109.803530s 2024-03-29T10:39:48.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fa382640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f92f40a9d70 0x7f92f4144050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:33948/0 (socket says 172.21.15.117:33948) 2024-03-29T10:39:48.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.541+0000 7f92fa382640 1 -- 172.21.15.117:0/2740099135 learned_addr learned my addr 172.21.15.117:0/2740099135 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:48.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 client.?.objecter ms_handle_connect 0x7f92f40a9d70 2024-03-29T10:39:48.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:48.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 client.?.objecter ms_handle_connect 0x7f92f41457f0 2024-03-29T10:39:48.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:48.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 client.?.objecter ms_handle_connect 0x7f92f413f650 2024-03-29T10:39:48.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:48.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492768592 0 0) 0x7f92f4140ac0 con 0x7f92f40a9d70 2024-03-29T10:39:48.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:48.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:48.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:48.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:39:48.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d8004eb0 con 0x7f92f40a9d70 2024-03-29T10:39:48.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407118035 0 0) 0x7f92f4143320 con 0x7f92f41457f0 2024-03-29T10:39:48.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f4140ac0 con 0x7f92f41457f0 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387261161 0 0) 0x7f92f41452b0 con 0x7f92f413f650 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f4143320 con 0x7f92f413f650 2024-03-29T10:39:48.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3271299456 0 0) 0x7f92d8004eb0 con 0x7f92f40a9d70 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92f41452b0 con 0x7f92f40a9d70 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f92f0001990 con 0x7f92f40a9d70 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.542+0000 7f92eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 10 monclient(hunting): dump: 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:48.544 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3570853836 0 0) 0x7f92f4140ac0 con 0x7f92f41457f0 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d8004eb0 con 0x7f92f41457f0 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f92ec0017b0 con 0x7f92f41457f0 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 10 monclient(hunting): dump: 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:48.545 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2144360635 0 0) 0x7f92f41452b0 con 0x7f92f40a9d70 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 >> v1:172.21.15.117:6790/0 conn(0x7f92f41457f0 legacy=0x7f92f41414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 >> v1:172.21.15.117:6789/0 conn(0x7f92f413f650 legacy=0x7f92f4133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 1 monclient: found mon.b 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:48.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f41434b0 con 0x7f92f40a9d70 2024-03-29T10:39:48.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:48.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:48.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:48.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92fc5e6640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:39:48.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92fc5e6640 10 monclient: _renew_subs 2024-03-29T10:39:48.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92fc5e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:48.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f4144d30 con 0x7f92f40a9d70 2024-03-29T10:39:48.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92fc5e6640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:48.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92fc5e6640 10 monclient: _renew_subs 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92fc5e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f4134580 con 0x7f92f40a9d70 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.543+0000 7f92fc5e6640 1 librados: init done 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.545+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92f0001af0 con 0x7f92f40a9d70 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.545+0000 7f92eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.545+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f92f00038c0 con 0x7f92f40a9d70 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.545+0000 7f92eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.545+0000 7f92eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.545+0000 7f92eaffd640 10 monclient: dump: 2024-03-29T10:39:48.548 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.545+0000 7f92f8b7f640 4 set_mon_vals no callback set 2024-03-29T10:39:48.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.545+0000 7f92fc5e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:48.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.546+0000 7f92fc5e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:48.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.546+0000 7f92fc5e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:48.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.546+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f4145030 con 0x7f92f40a9d70 2024-03-29T10:39:48.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.547+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f92f0003a20 con 0x7f92f40a9d70 2024-03-29T10:39:48.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.548+0000 7f92f9b81640 10 monclient: get_auth_request con 0x7f92d806dfb0 auth_method 0 2024-03-29T10:39:48.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.548+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f92f00849a0 con 0x7f92f40a9d70 2024-03-29T10:39:48.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.548+0000 7f92eaffd640 10 client.4402.objecter ms_dispatch 0x7f92f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:48.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.548+0000 7f92eaffd640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:48.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.548+0000 7f92eaffd640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:48.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.548+0000 7f92eaffd640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T10:39:48.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.548+0000 7f92eaffd640 10 client.4402.objecter ms_handle_connect 0x7f92d806dfb0 2024-03-29T10:39:48.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.554+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92f0050280 con 0x7f92f40a9d70 2024-03-29T10:39:48.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.554+0000 7f92eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:48.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.554+0000 7f92eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:48.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.921+0000 7f92fc5e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:48.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.922+0000 7f92fc5e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:48.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.922+0000 7f92fc5e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:39:48.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.922+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92f4075f10 con 0x7f92f40a9d70 2024-03-29T10:39:48.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.924+0000 7f92eaffd640 1 -- 172.21.15.117:0/2740099135 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f92f00521b0 con 0x7f92f40a9d70 2024-03-29T10:39:48.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.924+0000 7f92eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:48.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.924+0000 7f92eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:48.908 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:39:48.908 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:48.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.927+0000 7f92fc5e6640 10 librados: watch_flush enter 2024-03-29T10:39:48.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.927+0000 7f92fc5e6640 10 librados: watch_flush exit 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.927+0000 7f92fc5e6640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.927+0000 7f92fc5e6640 10 client.4402.objecter successfully canceled tick 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.928+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 >> v1:172.21.15.117:6816/376768171 conn(0x7f92d806dfb0 legacy=0x7f92d8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.928+0000 7f92fc5e6640 10 monclient: shutdown 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.928+0000 7f92fc5e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.928+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 >> v1:172.21.15.136:6789/0 conn(0x7f92f40a9d70 legacy=0x7f92f4144050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.928+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 shutdown_connections 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.928+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 >> 172.21.15.117:0/2740099135 conn(0x7f92f4066a00 msgr2=0x7f92f413a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.928+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 shutdown_connections 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.928+0000 7f92fc5e6640 1 -- 172.21.15.117:0/2740099135 wait complete. 2024-03-29T10:39:48.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:48.928+0000 7f92fc5e6640 1 librados: shutdown 2024-03-29T10:39:48.925 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:55.733 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.872+0000 7fb72cff6640 10 monclient: get_monmap_and_config 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.872+0000 7fb72cff6640 10 monclient: build_initial_monmap 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.872+0000 7fb72cff6640 10 monclient: monmap: 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:39:55.872967+0000 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:39:55.872967+0000 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:39:55.856 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:39:55.857 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.872+0000 7fb72cff6640 1 Processor -- start 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.872+0000 7fb72cff6640 1 -- start start 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.872+0000 7fb72cff6640 10 monclient: init 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.873+0000 7fb72cff6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.873+0000 7fb72cff6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.873+0000 7fb72cff6640 10 monclient(hunting): picked mon.noname-b con 0x7fb7281422b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.873+0000 7fb72cff6640 10 monclient(hunting): picked mon.noname-a con 0x7fb728142690 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.873+0000 7fb72cff6640 10 monclient(hunting): picked mon.noname-c con 0x7fb7280b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72659c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fb7281422b0 0x7fb72813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:48756/0 (socket says 172.21.15.117:48756) 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72659c640 1 -- 172.21.15.117:0/2509046927 learned_addr learned my addr 172.21.15.117:0/2509046927 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:55.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72cff6640 1 -- 172.21.15.117:0/2509046927 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb728142c80 con 0x7fb728142690 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72cff6640 1 -- 172.21.15.117:0/2509046927 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7280aae40 con 0x7fb7280b17d0 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72cff6640 1 -- 172.21.15.117:0/2509046927 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72813fbd0 con 0x7fb7281422b0 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72cff6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72cff6640 10 monclient(hunting): authenticate will time out at 1117.136426s 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72559a640 1 -- 172.21.15.117:0/2509046927 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370975215 0 0) 0x7fb72813fbd0 con 0x7fb7281422b0 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72559a640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72559a640 1 -- 172.21.15.117:0/2509046927 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb710003520 con 0x7fb7281422b0 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72559a640 1 -- 172.21.15.117:0/2509046927 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3907497536 0 0) 0x7fb710003520 con 0x7fb7281422b0 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72559a640 1 -- 172.21.15.117:0/2509046927 >> v1:172.21.15.136:6789/0 conn(0x7fb7280b17d0 legacy=0x7fb7280b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:55.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.874+0000 7fb72559a640 1 -- 172.21.15.117:0/2509046927 >> v1:172.21.15.117:6789/0 conn(0x7fb728142690 legacy=0x7fb7280b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 10 monclient: _finish_hunting 0 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 1 monclient: found mon.noname-b 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 1 -- 172.21.15.117:0/2509046927 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb728150930 con 0x7fb7281422b0 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 10 monclient: _finish_auth 0 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 10 monclient: _check_auth_tickets 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 1 -- 172.21.15.117:0/2509046927 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fb718001690 con 0x7fb7281422b0 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:55.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 10 monclient: dump: 2024-03-29T10:39:55.864 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:55.864 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:55.864 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:55.864 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:55.864 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72cff6640 5 monclient: authenticate success, global_id 4457 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72559a640 1 -- 172.21.15.117:0/2509046927 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7180018b0 con 0x7fb7281422b0 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.876+0000 7fb72cff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.877+0000 7fb72559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.877+0000 7fb72cff6640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.877+0000 7fb72cff6640 4 set_mon_vals no callback set 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.877+0000 7fb72559a640 1 -- 172.21.15.117:0/2509046927 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fb718001c40 con 0x7fb7281422b0 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.877+0000 7fb72559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:55.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.877+0000 7fb72559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.877+0000 7fb72559a640 10 monclient: dump: 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.878+0000 7fb72cff6640 10 monclient: shutdown 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.878+0000 7fb72cff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:55.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.878+0000 7fb72cff6640 1 -- 172.21.15.117:0/2509046927 >> v1:172.21.15.117:6790/0 conn(0x7fb7281422b0 legacy=0x7fb72813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.878+0000 7fb72cff6640 1 -- 172.21.15.117:0/2509046927 shutdown_connections 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.878+0000 7fb72cff6640 1 -- 172.21.15.117:0/2509046927 >> 172.21.15.117:0/2509046927 conn(0x7fb728066a00 msgr2=0x7fb728066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.878+0000 7fb72cff6640 1 -- 172.21.15.117:0/2509046927 shutdown_connections 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.878+0000 7fb72cff6640 1 -- 172.21.15.117:0/2509046927 wait complete. 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.879+0000 7fb72cff6640 10 monclient: build_initial_monmap 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.879+0000 7fb72cff6640 10 monclient: monmap: 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:39:55.867 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:39:55.868 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:55.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.879+0000 7fb72cff6640 1 librados: starting msgr at 2024-03-29T10:39:55.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.879+0000 7fb72cff6640 1 librados: starting objecter 2024-03-29T10:39:55.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.880+0000 7fb72cff6640 1 Processor -- start 2024-03-29T10:39:55.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.880+0000 7fb72cff6640 1 -- start start 2024-03-29T10:39:55.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.880+0000 7fb72cff6640 1 librados: setting wanted keys 2024-03-29T10:39:55.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.880+0000 7fb72cff6640 1 librados: calling monclient init 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.880+0000 7fb72cff6640 10 monclient: init 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.881+0000 7fb72cff6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.881+0000 7fb72cff6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.881+0000 7fb72cff6640 10 monclient(hunting): picked mon.noname-a con 0x7fb7280b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.881+0000 7fb72cff6640 10 monclient(hunting): picked mon.noname-b con 0x7fb7281422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.881+0000 7fb72cff6640 10 monclient(hunting): picked mon.noname-c con 0x7fb728142690 addr v1:172.21.15.117:6790/0 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.881+0000 7fb725d9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb7281422b0 0x7fb7280b54d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:36552/0 (socket says 172.21.15.117:36552) 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.881+0000 7fb725d9b640 1 -- 172.21.15.117:0/358806658 learned_addr learned my addr 172.21.15.117:0/358806658 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.881+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7280acd20 con 0x7fb7280b17d0 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7280ab540 con 0x7fb7281422b0 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 10 client.?.objecter ms_handle_connect 0x7fb728142690 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7280ad220 con 0x7fb728142690 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:55.877 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 10 client.?.objecter ms_handle_connect 0x7fb7281422b0 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb72cff6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb72cff6640 10 monclient(hunting): authenticate will time out at 1117.144704s 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164709484 0 0) 0x7fb7280ab540 con 0x7fb7281422b0 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb708004eb0 con 0x7fb7281422b0 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.882+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268190643 0 0) 0x7fb7280ad220 con 0x7fb728142690 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:55.878 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7280ab540 con 0x7fb728142690 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1950394138 0 0) 0x7fb7280ab540 con 0x7fb728142690 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7280ad220 con 0x7fb728142690 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fb720001b30 con 0x7fb728142690 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:55.879 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2645778508 0 0) 0x7fb7280ad220 con 0x7fb728142690 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.883+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 >> v1:172.21.15.136:6789/0 conn(0x7fb7281422b0 legacy=0x7fb7280b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 >> v1:172.21.15.117:6789/0 conn(0x7fb7280b17d0 legacy=0x7fb7280ac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 1 monclient: found mon.c 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7280ad3b0 con 0x7fb728142690 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:55.880 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb72cff6640 5 monclient: authenticate success, global_id 4460 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb72cff6640 10 monclient: _renew_subs 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb72cff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7280b5be0 con 0x7fb728142690 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb72cff6640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb72cff6640 10 monclient: _renew_subs 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb72cff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7280abee0 con 0x7fb728142690 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb72cff6640 1 librados: init done 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb720002040 con 0x7fb728142690 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb724d99640 4 set_mon_vals no callback set 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fb720003970 con 0x7fb728142690 2024-03-29T10:39:55.881 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.884+0000 7fb716ffd640 10 monclient: dump: 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.885+0000 7fb72cff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:55.882 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.885+0000 7fb72cff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.885+0000 7fb72cff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.885+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c005080 con 0x7fb728142690 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.886+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fb720019940 con 0x7fb728142690 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.887+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fb720083920 con 0x7fb728142690 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.887+0000 7fb72659c640 10 monclient: get_auth_request con 0x7fb70806daa0 auth_method 0 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.887+0000 7fb716ffd640 10 client.4460.objecter ms_dispatch 0x7fb728000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.887+0000 7fb716ffd640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.887+0000 7fb716ffd640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.887+0000 7fb716ffd640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.887+0000 7fb716ffd640 10 client.4460.objecter ms_handle_connect 0x7fb70806daa0 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.891+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb72004f200 con 0x7fb728142690 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.891+0000 7fb716ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:55.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:55.891+0000 7fb716ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:56.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.184+0000 7fb72cff6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:56.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.184+0000 7fb72cff6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:56.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.184+0000 7fb72cff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:39:56.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.184+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb70c005330 con 0x7fb728142690 2024-03-29T10:39:56.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.184+0000 7fb716ffd640 1 -- 172.21.15.117:0/358806658 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fb720051130 con 0x7fb728142690 2024-03-29T10:39:56.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.184+0000 7fb716ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:56.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.184+0000 7fb716ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:56.169 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:39:56.169 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 10 librados: watch_flush enter 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 10 librados: watch_flush exit 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 10 client.4460.objecter successfully canceled tick 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 >> v1:172.21.15.117:6816/376768171 conn(0x7fb70806daa0 legacy=0x7fb70806ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 10 monclient: shutdown 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 >> v1:172.21.15.117:6790/0 conn(0x7fb728142690 legacy=0x7fb7281e1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 shutdown_connections 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.187+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 >> 172.21.15.117:0/358806658 conn(0x7fb728066a00 msgr2=0x7fb728066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.188+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 shutdown_connections 2024-03-29T10:39:56.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.188+0000 7fb72cff6640 1 -- 172.21.15.117:0/358806658 wait complete. 2024-03-29T10:39:56.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:39:56.188+0000 7fb72cff6640 1 librados: shutdown 2024-03-29T10:39:56.185 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:02.992 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:03.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.142+0000 7fdeded3a640 10 monclient: get_monmap_and_config 2024-03-29T10:40:03.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.142+0000 7fdeded3a640 10 monclient: build_initial_monmap 2024-03-29T10:40:03.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.142+0000 7fdeded3a640 10 monclient: monmap: 2024-03-29T10:40:03.127 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:03.127 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:03.127 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:40:03.144356+0000 2024-03-29T10:40:03.127 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:40:03.144356+0000 2024-03-29T10:40:03.128 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:03.128 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:03.128 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:03.128 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:40:03.128 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:40:03.128 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:03.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.143+0000 7fdeded3a640 1 Processor -- start 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.143+0000 7fdeded3a640 1 -- start start 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.143+0000 7fdeded3a640 10 monclient: init 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.143+0000 7fdeded3a640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.143+0000 7fdeded3a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.143+0000 7fdeded3a640 10 monclient(hunting): picked mon.noname-b con 0x7fded80a9750 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.143+0000 7fdeded3a640 10 monclient(hunting): picked mon.noname-a con 0x7fded813e1e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.143+0000 7fdeded3a640 10 monclient(hunting): picked mon.noname-c con 0x7fded8071c30 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.143+0000 7fdeded3a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded80ab300 con 0x7fded813e1e0 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fdeded3a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded80ac2f0 con 0x7fded8071c30 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fdeded3a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded80ad8d0 con 0x7fded80a9750 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fdeded3a640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fdeded3a640 10 monclient(hunting): authenticate will time out at 1124.407137s 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fdedd2d7640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fded8071c30 0x7fded80ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:60058/0 (socket says 172.21.15.117:60058) 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fdedd2d7640 1 -- 172.21.15.117:0/18733341 learned_addr learned my addr 172.21.15.117:0/18733341 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fdedcad6640 1 --1- 172.21.15.117:0/18733341 >> v1:172.21.15.117:6790/0 conn(0x7fded80a9750 0x7fded813dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41790/0 (socket says 172.21.15.117:41790) 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566651560 0 0) 0x7fded80ad8d0 con 0x7fded80a9750 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec8003520 con 0x7fded80a9750 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889613985 0 0) 0x7fded80ac2f0 con 0x7fded8071c30 2024-03-29T10:40:03.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:03.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:03.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:03.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:40:03.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded80ad8d0 con 0x7fded8071c30 2024-03-29T10:40:03.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2625061064 0 0) 0x7fdec8003520 con 0x7fded80a9750 2024-03-29T10:40:03.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.144+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 >> v1:172.21.15.136:6789/0 conn(0x7fded8071c30 legacy=0x7fded80ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:03.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.145+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 >> v1:172.21.15.117:6789/0 conn(0x7fded813e1e0 legacy=0x7fded80ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:03.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:03.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 1 monclient: found mon.noname-b 2024-03-29T10:40:03.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:03.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:40:03.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded80ada60 con 0x7fded80a9750 2024-03-29T10:40:03.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdecc001690 con 0x7fded80a9750 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: dump: 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:03.140 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fdeded3a640 5 monclient: authenticate success, global_id 4463 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fdeded3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdecc001870 con 0x7fded80a9750 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 1 -- 172.21.15.117:0/18733341 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdecc001cc0 con 0x7fded80a9750 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:03.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fded77fe640 10 monclient: dump: 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fdeded3a640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.147+0000 7fdeded3a640 4 set_mon_vals no callback set 2024-03-29T10:40:03.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.148+0000 7fdeded3a640 10 monclient: shutdown 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.148+0000 7fdeded3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.148+0000 7fdeded3a640 1 -- 172.21.15.117:0/18733341 >> v1:172.21.15.117:6790/0 conn(0x7fded80a9750 legacy=0x7fded813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.148+0000 7fdeded3a640 1 -- 172.21.15.117:0/18733341 shutdown_connections 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.148+0000 7fdeded3a640 1 -- 172.21.15.117:0/18733341 >> 172.21.15.117:0/18733341 conn(0x7fded806b660 msgr2=0x7fded806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.148+0000 7fdeded3a640 1 -- 172.21.15.117:0/18733341 shutdown_connections 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.148+0000 7fdeded3a640 1 -- 172.21.15.117:0/18733341 wait complete. 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.149+0000 7fdeded3a640 10 monclient: build_initial_monmap 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.149+0000 7fdeded3a640 10 monclient: monmap: 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:40:03.143 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.149+0000 7fdeded3a640 1 librados: starting msgr at 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.149+0000 7fdeded3a640 1 librados: starting objecter 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.149+0000 7fdeded3a640 1 Processor -- start 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.149+0000 7fdeded3a640 1 -- start start 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.149+0000 7fdeded3a640 1 librados: setting wanted keys 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.149+0000 7fdeded3a640 1 librados: calling monclient init 2024-03-29T10:40:03.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.149+0000 7fdeded3a640 10 monclient: init 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 10 monclient(hunting): picked mon.noname-b con 0x7fded8071c30 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 10 monclient(hunting): picked mon.noname-c con 0x7fded80a9750 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 10 monclient(hunting): picked mon.noname-a con 0x7fded813e1e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded806ece0 con 0x7fded813e1e0 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded806efc0 con 0x7fded8071c30 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded806de60 con 0x7fded80a9750 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fdeded3a640 10 monclient(hunting): authenticate will time out at 1124.413735s 2024-03-29T10:40:03.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded7fff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fded80a9750 0x7fded8142340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41904/0 (socket says 172.21.15.117:41904) 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded7fff640 1 -- 172.21.15.117:0/1397762291 learned_addr learned my addr 172.21.15.117:0/1397762291 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded57fa640 10 client.?.objecter ms_handle_connect 0x7fded80a9750 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228349455 0 0) 0x7fded806de60 con 0x7fded80a9750 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded57fa640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.150+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdebc004eb0 con 0x7fded80a9750 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3202924749 0 0) 0x7fdebc004eb0 con 0x7fded80a9750 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fded806de60 con 0x7fded80a9750 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdec40017d0 con 0x7fded80a9750 2024-03-29T10:40:03.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 10 monclient(hunting): dump: 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:03.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3669325647 0 0) 0x7fded806de60 con 0x7fded80a9750 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 >> v1:172.21.15.136:6789/0 conn(0x7fded8071c30 legacy=0x7fded8140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.151+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 >> v1:172.21.15.117:6789/0 conn(0x7fded813e1e0 legacy=0x7fded806f3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fded57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fded57fa640 1 monclient: found mon.c 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fded57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fded57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded813ad50 con 0x7fded80a9750 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fded57fa640 10 monclient: _finish_auth 0 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fded57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fded57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fdeded3a640 5 monclient: authenticate success, global_id 4466 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fdeded3a640 10 monclient: _renew_subs 2024-03-29T10:40:03.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fdeded3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.152+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded806d9d0 con 0x7fded80a9750 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fdeded3a640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fdeded3a640 10 monclient: _renew_subs 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fdeded3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fded81e2fd0 con 0x7fded80a9750 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fdeded3a640 1 librados: init done 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdec4004860 con 0x7fded80a9750 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fded57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdec4004b70 con 0x7fded80a9750 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fded57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fded57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fded6ffd640 4 set_mon_vals no callback set 2024-03-29T10:40:03.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.153+0000 7fded57fa640 10 monclient: dump: 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.154+0000 7fdeded3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:03.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.154+0000 7fdeded3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:03.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.154+0000 7fdeded3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:03.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.154+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb8005080 con 0x7fded80a9750 2024-03-29T10:40:03.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.155+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fdec401ab40 con 0x7fded80a9750 2024-03-29T10:40:03.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.155+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fdec4084b50 con 0x7fded80a9750 2024-03-29T10:40:03.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.155+0000 7fdedcad6640 10 monclient: get_auth_request con 0x7fdebc06db80 auth_method 0 2024-03-29T10:40:03.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.155+0000 7fded57fa640 10 client.4466.objecter ms_dispatch 0x7fded8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:03.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.155+0000 7fded57fa640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:03.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.155+0000 7fded57fa640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:03.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.155+0000 7fded57fa640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T10:40:03.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.155+0000 7fded57fa640 10 client.4466.objecter ms_handle_connect 0x7fdebc06db80 2024-03-29T10:40:03.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.160+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdec4050260 con 0x7fded80a9750 2024-03-29T10:40:03.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.160+0000 7fded57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:03.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.160+0000 7fded57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:03.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.447+0000 7fdeded3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:03.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.447+0000 7fdeded3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:03.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.447+0000 7fdeded3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:03.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.447+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeb8005600 con 0x7fded80a9750 2024-03-29T10:40:03.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.447+0000 7fded57fa640 1 -- 172.21.15.117:0/1397762291 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fdec4052360 con 0x7fded80a9750 2024-03-29T10:40:03.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.447+0000 7fded57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:03.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.447+0000 7fded57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:03.433 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:40:03.433 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.450+0000 7fdeded3a640 10 librados: watch_flush enter 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.450+0000 7fdeded3a640 10 librados: watch_flush exit 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.450+0000 7fdeded3a640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.450+0000 7fdeded3a640 10 client.4466.objecter successfully canceled tick 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.450+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 >> v1:172.21.15.117:6816/376768171 conn(0x7fdebc06db80 legacy=0x7fdebc070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.451+0000 7fdeded3a640 10 monclient: shutdown 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.451+0000 7fdeded3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.451+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 >> v1:172.21.15.117:6790/0 conn(0x7fded80a9750 legacy=0x7fded8142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.451+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 shutdown_connections 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.451+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 >> 172.21.15.117:0/1397762291 conn(0x7fded806b660 msgr2=0x7fded80adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.451+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 shutdown_connections 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.451+0000 7fdeded3a640 1 -- 172.21.15.117:0/1397762291 wait complete. 2024-03-29T10:40:03.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:03.451+0000 7fdeded3a640 1 librados: shutdown 2024-03-29T10:40:03.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:10.256 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:10.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.418+0000 7f246bfaf640 10 monclient: get_monmap_and_config 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.418+0000 7f246bfaf640 10 monclient: build_initial_monmap 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.418+0000 7f246bfaf640 10 monclient: monmap: 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:40:10.419847+0000 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:40:10.419847+0000 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:40:10.403 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:10.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.420+0000 7f246bfaf640 1 Processor -- start 2024-03-29T10:40:10.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.420+0000 7f246bfaf640 1 -- start start 2024-03-29T10:40:10.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.420+0000 7f246bfaf640 10 monclient: init 2024-03-29T10:40:10.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.421+0000 7f246bfaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:10.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.421+0000 7f246bfaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:10.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.421+0000 7f246bfaf640 10 monclient(hunting): picked mon.noname-b con 0x7f24641457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:10.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.421+0000 7f246bfaf640 10 monclient(hunting): picked mon.noname-a con 0x7f246413f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:10.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.421+0000 7f246bfaf640 10 monclient(hunting): picked mon.noname-c con 0x7f24640a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:10.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.421+0000 7f246bfaf640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24640aaef0 con 0x7f246413f650 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.421+0000 7f246bfaf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2464145bd0 con 0x7f24640a9d70 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.423+0000 7f246bfaf640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2464145eb0 con 0x7f24641457f0 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.423+0000 7f246bfaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.423+0000 7f246bfaf640 10 monclient(hunting): authenticate will time out at 1131.685602s 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.423+0000 7f2469d4b640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f24641457f0 0x7f246413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:37586/0 (socket says 172.21.15.117:37586) 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.423+0000 7f2469d4b640 1 -- 172.21.15.117:0/907312411 learned_addr learned my addr 172.21.15.117:0/907312411 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.424+0000 7f2468d49640 1 -- 172.21.15.117:0/907312411 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298764929 0 0) 0x7f2464145eb0 con 0x7f24641457f0 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.424+0000 7f2468d49640 10 monclient(hunting): _negotiate 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.424+0000 7f2468d49640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.424+0000 7f2468d49640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.424+0000 7f2468d49640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.424+0000 7f2468d49640 1 -- 172.21.15.117:0/907312411 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2450003520 con 0x7f24641457f0 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.424+0000 7f2468d49640 1 -- 172.21.15.117:0/907312411 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2567516433 0 0) 0x7f2450003520 con 0x7f24641457f0 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.424+0000 7f2468d49640 1 -- 172.21.15.117:0/907312411 >> v1:172.21.15.136:6789/0 conn(0x7f24640a9d70 legacy=0x7f246414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.424+0000 7f2468d49640 1 -- 172.21.15.117:0/907312411 >> v1:172.21.15.117:6789/0 conn(0x7f246413f650 legacy=0x7f24640b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 10 monclient: _finish_hunting 0 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 1 monclient: found mon.noname-b 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 1 -- 172.21.15.117:0/907312411 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2464146340 con 0x7f24641457f0 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 10 monclient: _finish_auth 0 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 10 monclient: _check_auth_tickets 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 1 -- 172.21.15.117:0/907312411 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2460001690 con 0x7f24641457f0 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f2468d49640 10 monclient: dump: 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f246bfaf640 5 monclient: authenticate success, global_id 4469 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.427+0000 7f246bfaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.428+0000 7f2468d49640 1 -- 172.21.15.117:0/907312411 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24600019a0 con 0x7f24641457f0 2024-03-29T10:40:10.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.428+0000 7f2468d49640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.428+0000 7f2468d49640 1 -- 172.21.15.117:0/907312411 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2460001d30 con 0x7f24641457f0 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.428+0000 7f2468d49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.428+0000 7f2468d49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.428+0000 7f2468d49640 10 monclient: dump: 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:10.414 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:10.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.428+0000 7f246bfaf640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:10.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.428+0000 7f246bfaf640 4 set_mon_vals no callback set 2024-03-29T10:40:10.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.430+0000 7f246bfaf640 10 monclient: shutdown 2024-03-29T10:40:10.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.430+0000 7f246bfaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:10.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.430+0000 7f246bfaf640 1 -- 172.21.15.117:0/907312411 >> v1:172.21.15.117:6790/0 conn(0x7f24641457f0 legacy=0x7f246413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.430+0000 7f246bfaf640 1 -- 172.21.15.117:0/907312411 shutdown_connections 2024-03-29T10:40:10.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.430+0000 7f246bfaf640 1 -- 172.21.15.117:0/907312411 >> 172.21.15.117:0/907312411 conn(0x7f2464066a00 msgr2=0x7f2464066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:10.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.433+0000 7f246bfaf640 1 -- 172.21.15.117:0/907312411 shutdown_connections 2024-03-29T10:40:10.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.433+0000 7f246bfaf640 1 -- 172.21.15.117:0/907312411 wait complete. 2024-03-29T10:40:10.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.434+0000 7f246bfaf640 10 monclient: build_initial_monmap 2024-03-29T10:40:10.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.434+0000 7f246bfaf640 10 monclient: monmap: 2024-03-29T10:40:10.421 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:10.421 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:10.421 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:40:10.421 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.435+0000 7f246bfaf640 1 librados: starting msgr at 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.435+0000 7f246bfaf640 1 librados: starting objecter 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.435+0000 7f246bfaf640 1 Processor -- start 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.435+0000 7f246bfaf640 1 -- start start 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.435+0000 7f246bfaf640 1 librados: setting wanted keys 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.435+0000 7f246bfaf640 1 librados: calling monclient init 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.435+0000 7f246bfaf640 10 monclient: init 2024-03-29T10:40:10.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.436+0000 7f246bfaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:10.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.436+0000 7f246bfaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:10.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246bfaf640 10 monclient(hunting): picked mon.noname-b con 0x7f24640a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:10.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246bfaf640 10 monclient(hunting): picked mon.noname-c con 0x7f246413f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:10.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246bfaf640 10 monclient(hunting): picked mon.noname-a con 0x7f24641457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:10.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246bfaf640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24641e84a0 con 0x7f24641457f0 2024-03-29T10:40:10.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246bfaf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2464135a60 con 0x7f24640a9d70 2024-03-29T10:40:10.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246bfaf640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2464142d50 con 0x7f246413f650 2024-03-29T10:40:10.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246bfaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:10.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246bfaf640 10 monclient(hunting): authenticate will time out at 1131.700517s 2024-03-29T10:40:10.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246954a640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f246413f650 0x7f24641e7680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:38140/0 (socket says 172.21.15.117:38140) 2024-03-29T10:40:10.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f246954a640 1 -- 172.21.15.117:0/668141386 learned_addr learned my addr 172.21.15.117:0/668141386 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:10.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f245a7fc640 10 client.?.objecter ms_handle_connect 0x7f246413f650 2024-03-29T10:40:10.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.438+0000 7f245a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:10.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847315494 0 0) 0x7f2464142d50 con 0x7f246413f650 2024-03-29T10:40:10.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:10.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:10.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:10.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:40:10.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2448004eb0 con 0x7f246413f650 2024-03-29T10:40:10.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 client.?.objecter ms_handle_connect 0x7f24640a9d70 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2458042384 0 0) 0x7f2448004eb0 con 0x7f246413f650 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2464142d50 con 0x7f246413f650 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2454001960 con 0x7f246413f650 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:10.427 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:10.428 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:10.428 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:10.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.442+0000 7f245a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:10.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596185375 0 0) 0x7f2464135a60 con 0x7f24640a9d70 2024-03-29T10:40:10.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:10.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:10.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:10.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:40:10.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2448004eb0 con 0x7f24640a9d70 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 356984842 0 0) 0x7f2464142d50 con 0x7f246413f650 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 >> v1:172.21.15.136:6789/0 conn(0x7f24640a9d70 legacy=0x7f2464141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 >> v1:172.21.15.117:6789/0 conn(0x7f24641457f0 legacy=0x7f24641e7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 1 monclient: found mon.c 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246406eef0 con 0x7f246413f650 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f245a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f246bfaf640 5 monclient: authenticate success, global_id 4472 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f246bfaf640 10 monclient: _renew_subs 2024-03-29T10:40:10.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f246bfaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:10.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246406f1d0 con 0x7f246413f650 2024-03-29T10:40:10.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f246bfaf640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:10.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f246bfaf640 10 monclient: _renew_subs 2024-03-29T10:40:10.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f246bfaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:10.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2464071fc0 con 0x7f246413f650 2024-03-29T10:40:10.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.443+0000 7f246bfaf640 1 librados: init done 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.445+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2454001cc0 con 0x7f246413f650 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.445+0000 7f245a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.445+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f2454003a40 con 0x7f246413f650 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.445+0000 7f245a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.445+0000 7f245a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.445+0000 7f245a7fc640 10 monclient: dump: 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:10.432 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.445+0000 7f245bfff640 4 set_mon_vals no callback set 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.445+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f2454003c60 con 0x7f246413f650 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.446+0000 7f246bfaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.446+0000 7f246bfaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.446+0000 7f246bfaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.446+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c005080 con 0x7f246413f650 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.448+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f2454084ba0 con 0x7f246413f650 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.448+0000 7f245a7fc640 10 client.4472.objecter ms_dispatch 0x7f2464000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.448+0000 7f245a7fc640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:10.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.448+0000 7f245a7fc640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:10.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.448+0000 7f245a7fc640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T10:40:10.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.448+0000 7f2469d4b640 10 monclient: get_auth_request con 0x7f244806dd40 auth_method 0 2024-03-29T10:40:10.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.448+0000 7f245a7fc640 10 client.4472.objecter ms_handle_connect 0x7f244806dd40 2024-03-29T10:40:10.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.456+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24540505e0 con 0x7f246413f650 2024-03-29T10:40:10.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.456+0000 7f245a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:10.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.456+0000 7f245a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:10.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.772+0000 7f246bfaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:10.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.772+0000 7f246bfaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:10.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.772+0000 7f246bfaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:10.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.772+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f244c005330 con 0x7f246413f650 2024-03-29T10:40:10.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.773+0000 7f245a7fc640 1 -- 172.21.15.117:0/668141386 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f2454052510 con 0x7f246413f650 2024-03-29T10:40:10.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.773+0000 7f245a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:10.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.773+0000 7f245a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:10.757 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:40:10.758 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:10.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 10 librados: watch_flush enter 2024-03-29T10:40:10.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 10 librados: watch_flush exit 2024-03-29T10:40:10.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T10:40:10.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 10 client.4472.objecter successfully canceled tick 2024-03-29T10:40:10.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 >> v1:172.21.15.117:6816/376768171 conn(0x7f244806dd40 legacy=0x7f2448070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 10 monclient: shutdown 2024-03-29T10:40:10.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:10.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 >> v1:172.21.15.117:6790/0 conn(0x7f246413f650 legacy=0x7f24641e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 shutdown_connections 2024-03-29T10:40:10.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.777+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 >> 172.21.15.117:0/668141386 conn(0x7f2464066a00 msgr2=0x7f246413b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:10.763 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.779+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 shutdown_connections 2024-03-29T10:40:10.763 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.779+0000 7f246bfaf640 1 -- 172.21.15.117:0/668141386 wait complete. 2024-03-29T10:40:10.763 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:10.779+0000 7f246bfaf640 1 librados: shutdown 2024-03-29T10:40:10.780 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:17.588 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:17.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.748+0000 7fe35ad4a640 10 monclient: get_monmap_and_config 2024-03-29T10:40:17.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.748+0000 7fe35ad4a640 10 monclient: build_initial_monmap 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.748+0000 7fe35ad4a640 10 monclient: monmap: 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:40:17.749604+0000 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:40:17.749604+0000 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:40:17.733 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:17.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.750+0000 7fe35ad4a640 1 Processor -- start 2024-03-29T10:40:17.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.750+0000 7fe35ad4a640 1 -- start start 2024-03-29T10:40:17.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.750+0000 7fe35ad4a640 10 monclient: init 2024-03-29T10:40:17.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.751+0000 7fe35ad4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:17.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.751+0000 7fe35ad4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.751+0000 7fe35ad4a640 10 monclient(hunting): picked mon.noname-c con 0x7fe34c006ad0 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.751+0000 7fe35ad4a640 10 monclient(hunting): picked mon.noname-a con 0x7fe34c036870 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.752+0000 7fe35ad4a640 10 monclient(hunting): picked mon.noname-b con 0x7fe34c036d00 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.752+0000 7fe35ad4a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c037150 con 0x7fe34c036870 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.752+0000 7fe35ad4a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c033930 con 0x7fe34c006ad0 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.752+0000 7fe35ad4a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c033c10 con 0x7fe34c036d00 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.752+0000 7fe35ad4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.752+0000 7fe35ad4a640 10 monclient(hunting): authenticate will time out at 1139.014245s 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.752+0000 7fe35a549640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe34c036d00 0x7fe34c0d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40404/0 (socket says 172.21.15.117:40404) 2024-03-29T10:40:17.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.752+0000 7fe35a549640 1 -- 172.21.15.117:0/3424438930 learned_addr learned my addr 172.21.15.117:0/3424438930 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:17.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.755+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925440823 0 0) 0x7fe34c033c10 con 0x7fe34c036d00 2024-03-29T10:40:17.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.755+0000 7fe358d46640 10 monclient(hunting): _negotiate 2024-03-29T10:40:17.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.755+0000 7fe358d46640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:17.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.755+0000 7fe358d46640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:17.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.755+0000 7fe358d46640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:40:17.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.755+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe340003520 con 0x7fe34c036d00 2024-03-29T10:40:17.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258769837 0 0) 0x7fe34c033930 con 0x7fe34c006ad0 2024-03-29T10:40:17.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient(hunting): _negotiate 2024-03-29T10:40:17.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:17.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:17.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient(hunting): my global_id is 4414 2024-03-29T10:40:17.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe34c033c10 con 0x7fe34c006ad0 2024-03-29T10:40:17.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 590314279 0 0) 0x7fe340003520 con 0x7fe34c036d00 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 >> v1:172.21.15.136:6789/0 conn(0x7fe34c006ad0 legacy=0x7fe34c0322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 >> v1:172.21.15.117:6789/0 conn(0x7fe34c036870 legacy=0x7fe34c0d4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient: _finish_hunting 0 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 1 monclient: found mon.noname-b 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c0308c0 con 0x7fe34c036d00 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient: _finish_auth 0 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient: _check_auth_tickets 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe35404f730 con 0x7fe34c036d00 2024-03-29T10:40:17.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient: dump: 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:17.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe35ad4a640 5 monclient: authenticate success, global_id 4475 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe35ad4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe354075f10 con 0x7fe34c036d00 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe358d46640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe35ad4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.758+0000 7fe35ad4a640 4 set_mon_vals no callback set 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.759+0000 7fe358d46640 1 -- 172.21.15.117:0/3424438930 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe3540758b0 con 0x7fe34c036d00 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.759+0000 7fe358d46640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.759+0000 7fe358d46640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.759+0000 7fe358d46640 10 monclient: dump: 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:17.745 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:17.746 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:17.746 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:17.746 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:17.746 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:17.746 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:17.746 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:17.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.759+0000 7fe35ad4a640 10 monclient: shutdown 2024-03-29T10:40:17.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.759+0000 7fe35ad4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:17.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.759+0000 7fe35ad4a640 1 -- 172.21.15.117:0/3424438930 >> v1:172.21.15.117:6790/0 conn(0x7fe34c036d00 legacy=0x7fe34c0d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.760+0000 7fe35ad4a640 1 -- 172.21.15.117:0/3424438930 shutdown_connections 2024-03-29T10:40:17.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.760+0000 7fe35ad4a640 1 -- 172.21.15.117:0/3424438930 >> 172.21.15.117:0/3424438930 conn(0x7fe34c0058f0 msgr2=0x7fe34c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:17.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.760+0000 7fe35ad4a640 1 -- 172.21.15.117:0/3424438930 shutdown_connections 2024-03-29T10:40:17.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.760+0000 7fe35ad4a640 1 -- 172.21.15.117:0/3424438930 wait complete. 2024-03-29T10:40:17.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.761+0000 7fe35ad4a640 10 monclient: build_initial_monmap 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.761+0000 7fe35ad4a640 10 monclient: monmap: 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.762+0000 7fe35ad4a640 1 librados: starting msgr at 2024-03-29T10:40:17.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.762+0000 7fe35ad4a640 1 librados: starting objecter 2024-03-29T10:40:17.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.762+0000 7fe35ad4a640 1 Processor -- start 2024-03-29T10:40:17.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.762+0000 7fe35ad4a640 1 -- start start 2024-03-29T10:40:17.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.762+0000 7fe35ad4a640 1 librados: setting wanted keys 2024-03-29T10:40:17.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.762+0000 7fe35ad4a640 1 librados: calling monclient init 2024-03-29T10:40:17.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.762+0000 7fe35ad4a640 10 monclient: init 2024-03-29T10:40:17.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:17.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:17.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 10 monclient(hunting): picked mon.noname-a con 0x7fe34c006ad0 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:17.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 10 monclient(hunting): picked mon.noname-b con 0x7fe34c036870 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 10 monclient(hunting): picked mon.noname-c con 0x7fe34c036d00 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c02eaf0 con 0x7fe34c006ad0 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c02ed60 con 0x7fe34c036870 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c02f040 con 0x7fe34c036d00 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.763+0000 7fe35ad4a640 10 monclient(hunting): authenticate will time out at 1139.025494s 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.764+0000 7fe35a549640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe34c036d00 0x7fe34c185270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40910/0 (socket says 172.21.15.117:40910) 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.764+0000 7fe35a549640 1 -- 172.21.15.117:0/1207626958 learned_addr learned my addr 172.21.15.117:0/1207626958 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.765+0000 7fe35a549640 1 -- 172.21.15.117:0/1207626958 >> v1:172.21.15.117:6790/0 conn(0x7fe34c036d00 legacy=0x7fe34c185270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.765+0000 7fe35a549640 1 -- 172.21.15.117:0/1207626958 >> v1:172.21.15.117:6790/0 conn(0x7fe34c036d00 legacy=0x7fe34c185270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.765+0000 7fe35a549640 1 --1- 172.21.15.117:0/1207626958 >> v1:172.21.15.117:6790/0 conn(0x7fe34c036d00 0x7fe34c185270 :-1 s=OPENED pgs=582700 cs=1 l=1).handle_message read tag failed 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.765+0000 7fe35a549640 1 --1- 172.21.15.117:0/1207626958 >> v1:172.21.15.117:6790/0 conn(0x7fe34c036d00 0x7fe34c185270 :-1 s=OPENED pgs=582700 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.765+0000 7fe34a7fc640 10 client.?.objecter ms_handle_connect 0x7fe34c036d00 2024-03-29T10:40:17.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.765+0000 7fe34a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:17.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.765+0000 7fe34a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.117:6790/0 2024-03-29T10:40:17.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.768+0000 7fe34a7fc640 10 client.?.objecter ms_handle_connect 0x7fe34c006ad0 2024-03-29T10:40:17.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.768+0000 7fe34a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:17.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.770+0000 7fe34a7fc640 10 client.?.objecter ms_handle_connect 0x7fe34c036870 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.770+0000 7fe34a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.770+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998811727 0 0) 0x7fe34c02eaf0 con 0x7fe34c006ad0 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.770+0000 7fe34a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.770+0000 7fe34a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.770+0000 7fe34a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.770+0000 7fe34a7fc640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.770+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe338004eb0 con 0x7fe34c006ad0 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157365865 0 0) 0x7fe34c02ed60 con 0x7fe34c036870 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:17.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient(hunting): my global_id is 4417 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe34c02eaf0 con 0x7fe34c036870 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 882777567 0 0) 0x7fe338004eb0 con 0x7fe34c006ad0 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe34c02ed60 con 0x7fe34c006ad0 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe350001950 con 0x7fe34c006ad0 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:17.759 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 762340876 0 0) 0x7fe34c02eaf0 con 0x7fe34c036870 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe338004eb0 con 0x7fe34c036870 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1311421805 0 0) 0x7fe34c02ed60 con 0x7fe34c006ad0 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 >> v1:172.21.15.117:6790/0 conn(0x7fe34c036d00 legacy=0x7fe34c185270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 >> v1:172.21.15.136:6789/0 conn(0x7fe34c036870 legacy=0x7fe34c0ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 monclient: found mon.a 2024-03-29T10:40:17.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c02f320 con 0x7fe34c006ad0 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.771+0000 7fe34a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.772+0000 7fe35ad4a640 5 monclient: authenticate success, global_id 4431 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.772+0000 7fe35ad4a640 10 monclient: _renew_subs 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.772+0000 7fe35ad4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.772+0000 7fe35ad4a640 1 -- 172.21.15.117:0/1207626958 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c0cc770 con 0x7fe34c006ad0 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.772+0000 7fe35ad4a640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.772+0000 7fe35ad4a640 10 monclient: _renew_subs 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.772+0000 7fe35ad4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.772+0000 7fe35ad4a640 1 -- 172.21.15.117:0/1207626958 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c02fc90 con 0x7fe34c006ad0 2024-03-29T10:40:17.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.772+0000 7fe35ad4a640 1 librados: init done 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.773+0000 7fe33ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.773+0000 7fe33ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.773+0000 7fe33ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.773+0000 7fe33ffff640 1 -- 172.21.15.117:0/1207626958 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe334005080 con 0x7fe34c006ad0 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.774+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe350003610 con 0x7fe34c006ad0 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.774+0000 7fe34a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.774+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe350003960 con 0x7fe34c006ad0 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.774+0000 7fe34a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.774+0000 7fe34a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.774+0000 7fe34a7fc640 10 monclient: dump: 2024-03-29T10:40:17.762 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:17.763 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:17.763 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:17.763 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:17.763 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:17.763 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:17.763 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:17.763 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:17.763 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:17.763 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:17.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.776+0000 7fe34b7fe640 4 set_mon_vals no callback set 2024-03-29T10:40:17.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.779+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fe350003ac0 con 0x7fe34c006ad0 2024-03-29T10:40:17.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.779+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fe350053590 con 0x7fe34c006ad0 2024-03-29T10:40:17.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.779+0000 7fe34a7fc640 10 client.4431.objecter ms_dispatch 0x7fe354000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:17.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.779+0000 7fe34a7fc640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:17.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.779+0000 7fe34a7fc640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:17.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.780+0000 7fe34a7fc640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-29T10:40:17.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.780+0000 7fe359547640 10 monclient: get_auth_request con 0x7fe33806dd70 auth_method 0 2024-03-29T10:40:17.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.780+0000 7fe34a7fc640 10 client.4431.objecter ms_handle_connect 0x7fe33806dd70 2024-03-29T10:40:17.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.783+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe35004f830 con 0x7fe34c006ad0 2024-03-29T10:40:17.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.783+0000 7fe34a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:17.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:17.783+0000 7fe34a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:18.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.100+0000 7fe33ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:18.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.100+0000 7fe33ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:18.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.100+0000 7fe33ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:40:18.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.100+0000 7fe33ffff640 1 -- 172.21.15.117:0/1207626958 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe334005330 con 0x7fe34c006ad0 2024-03-29T10:40:18.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.101+0000 7fe34a7fc640 1 -- 172.21.15.117:0/1207626958 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fe35004f9d0 con 0x7fe34c006ad0 2024-03-29T10:40:18.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.101+0000 7fe34a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:18.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.101+0000 7fe34a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:18.086 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:40:18.086 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:18.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.106+0000 7fe33ffff640 10 librados: watch_flush enter 2024-03-29T10:40:18.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.106+0000 7fe33ffff640 10 librados: watch_flush exit 2024-03-29T10:40:18.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.106+0000 7fe33ffff640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-29T10:40:18.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.106+0000 7fe33ffff640 10 client.4431.objecter successfully canceled tick 2024-03-29T10:40:18.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.107+0000 7fe33ffff640 1 -- 172.21.15.117:0/1207626958 >> v1:172.21.15.117:6816/376768171 conn(0x7fe33806dd70 legacy=0x7fe338070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:18.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.107+0000 7fe33ffff640 10 monclient: shutdown 2024-03-29T10:40:18.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.107+0000 7fe33ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:18.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.107+0000 7fe33ffff640 1 -- 172.21.15.117:0/1207626958 >> v1:172.21.15.117:6789/0 conn(0x7fe34c006ad0 legacy=0x7fe34c0ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:18.093 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.109+0000 7fe33ffff640 1 -- 172.21.15.117:0/1207626958 shutdown_connections 2024-03-29T10:40:18.093 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.109+0000 7fe33ffff640 1 -- 172.21.15.117:0/1207626958 >> 172.21.15.117:0/1207626958 conn(0x7fe34c0058f0 msgr2=0x7fe34c038e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:18.093 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.109+0000 7fe33ffff640 1 -- 172.21.15.117:0/1207626958 shutdown_connections 2024-03-29T10:40:18.093 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.109+0000 7fe33ffff640 1 -- 172.21.15.117:0/1207626958 wait complete. 2024-03-29T10:40:18.093 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:18.110+0000 7fe33ffff640 1 librados: shutdown 2024-03-29T10:40:18.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:24.916 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.080+0000 7fd1d21d0640 10 monclient: get_monmap_and_config 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.080+0000 7fd1d21d0640 10 monclient: build_initial_monmap 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.080+0000 7fd1d21d0640 10 monclient: monmap: 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:40:25.081276+0000 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:40:25.081276+0000 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:25.064 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:25.065 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:40:25.065 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:40:25.065 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:25.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.081+0000 7fd1d21d0640 1 Processor -- start 2024-03-29T10:40:25.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.081+0000 7fd1d21d0640 1 -- start start 2024-03-29T10:40:25.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.081+0000 7fd1d21d0640 10 monclient: init 2024-03-29T10:40:25.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.082+0000 7fd1d21d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:25.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.082+0000 7fd1d21d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:25.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1d21d0640 10 monclient(hunting): picked mon.noname-c con 0x7fd1cc1457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:25.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1d21d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd1cc13f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:25.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1d21d0640 10 monclient(hunting): picked mon.noname-a con 0x7fd1cc0a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:25.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1d21d0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0aaef0 con 0x7fd1cc0a9d70 2024-03-29T10:40:25.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1d21d0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc145bd0 con 0x7fd1cc1457f0 2024-03-29T10:40:25.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1d21d0640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc145eb0 con 0x7fd1cc13f650 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1d21d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1d21d0640 10 monclient(hunting): authenticate will time out at 1146.346734s 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1caffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd1cc13f650 0x7fd1cc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:42560/0 (socket says 172.21.15.117:42560) 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.084+0000 7fd1caffd640 1 -- 172.21.15.117:0/3112381046 learned_addr learned my addr 172.21.15.117:0/3112381046 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.085+0000 7fd1ca7fc640 1 -- 172.21.15.117:0/3112381046 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639311672 0 0) 0x7fd1cc145eb0 con 0x7fd1cc13f650 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.085+0000 7fd1ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.085+0000 7fd1ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.085+0000 7fd1ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.085+0000 7fd1ca7fc640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.085+0000 7fd1ca7fc640 1 -- 172.21.15.117:0/3112381046 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b4003520 con 0x7fd1cc13f650 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.085+0000 7fd1ca7fc640 1 -- 172.21.15.117:0/3112381046 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 14837619 0 0) 0x7fd1b4003520 con 0x7fd1cc13f650 2024-03-29T10:40:25.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.085+0000 7fd1ca7fc640 1 -- 172.21.15.117:0/3112381046 >> v1:172.21.15.136:6789/0 conn(0x7fd1cc1457f0 legacy=0x7fd1cc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:25.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.086+0000 7fd1ca7fc640 1 -- 172.21.15.117:0/3112381046 >> v1:172.21.15.117:6789/0 conn(0x7fd1cc0a9d70 legacy=0x7fd1cc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:25.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:25.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 1 monclient: found mon.noname-b 2024-03-29T10:40:25.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:25.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:40:25.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 1 -- 172.21.15.117:0/3112381046 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc146340 con 0x7fd1cc13f650 2024-03-29T10:40:25.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:25.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 1 -- 172.21.15.117:0/3112381046 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd1bc001690 con 0x7fd1cc13f650 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.088+0000 7fd1ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.089+0000 7fd1ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.089+0000 7fd1ca7fc640 10 monclient: dump: 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:25.074 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.089+0000 7fd1ca7fc640 1 -- 172.21.15.117:0/3112381046 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1bc0019a0 con 0x7fd1cc13f650 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.089+0000 7fd1ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.089+0000 7fd1ca7fc640 1 -- 172.21.15.117:0/3112381046 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd1bc001d30 con 0x7fd1cc13f650 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.089+0000 7fd1ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.089+0000 7fd1ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.089+0000 7fd1ca7fc640 10 monclient: dump: 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:25.075 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:25.076 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.090+0000 7fd1d21d0640 5 monclient: authenticate success, global_id 4478 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.090+0000 7fd1d21d0640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.090+0000 7fd1d21d0640 4 set_mon_vals no callback set 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.091+0000 7fd1d21d0640 10 monclient: shutdown 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.091+0000 7fd1d21d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.091+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3112381046 >> v1:172.21.15.117:6790/0 conn(0x7fd1cc13f650 legacy=0x7fd1cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.092+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3112381046 shutdown_connections 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.092+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3112381046 >> 172.21.15.117:0/3112381046 conn(0x7fd1cc066a00 msgr2=0x7fd1cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.092+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3112381046 shutdown_connections 2024-03-29T10:40:25.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.093+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3112381046 wait complete. 2024-03-29T10:40:25.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.094+0000 7fd1d21d0640 10 monclient: build_initial_monmap 2024-03-29T10:40:25.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.094+0000 7fd1d21d0640 10 monclient: monmap: 2024-03-29T10:40:25.078 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:25.078 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:25.078 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:40:25.078 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:40:25.078 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:25.079 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:25.079 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:25.079 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:40:25.079 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:40:25.079 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:25.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.094+0000 7fd1d21d0640 1 librados: starting msgr at 2024-03-29T10:40:25.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.094+0000 7fd1d21d0640 1 librados: starting objecter 2024-03-29T10:40:25.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.095+0000 7fd1d21d0640 1 Processor -- start 2024-03-29T10:40:25.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.095+0000 7fd1d21d0640 1 -- start start 2024-03-29T10:40:25.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.095+0000 7fd1d21d0640 1 librados: setting wanted keys 2024-03-29T10:40:25.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.095+0000 7fd1d21d0640 1 librados: calling monclient init 2024-03-29T10:40:25.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.095+0000 7fd1d21d0640 10 monclient: init 2024-03-29T10:40:25.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.096+0000 7fd1d21d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:25.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.096+0000 7fd1d21d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:25.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.097+0000 7fd1d21d0640 10 monclient(hunting): picked mon.noname-a con 0x7fd1cc0a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:25.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.097+0000 7fd1d21d0640 10 monclient(hunting): picked mon.noname-c con 0x7fd1cc1457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:25.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.097+0000 7fd1d21d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd1cc071e50 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:25.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.097+0000 7fd1d21d0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0700a0 con 0x7fd1cc0a9d70 2024-03-29T10:40:25.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.097+0000 7fd1d21d0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc1df0f0 con 0x7fd1cc071e50 2024-03-29T10:40:25.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.097+0000 7fd1d21d0640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc1df280 con 0x7fd1cc1457f0 2024-03-29T10:40:25.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.097+0000 7fd1d21d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:25.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.097+0000 7fd1d21d0640 10 monclient(hunting): authenticate will time out at 1146.359186s 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.098+0000 7fd1cb7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd1cc0a9d70 0x7fd1cc1de9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:52587/0 (socket says 172.21.15.117:52587) 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.098+0000 7fd1cb7fe640 1 -- 172.21.15.117:0/3795263482 learned_addr learned my addr 172.21.15.117:0/3795263482 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.099+0000 7fd1b3fff640 10 client.?.objecter ms_handle_connect 0x7fd1cc0a9d70 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.099+0000 7fd1b3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.100+0000 7fd1b3fff640 10 client.?.objecter ms_handle_connect 0x7fd1cc071e50 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.100+0000 7fd1b3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.101+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788418928 0 0) 0x7fd1cc1df0f0 con 0x7fd1cc071e50 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.101+0000 7fd1b3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.101+0000 7fd1b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.101+0000 7fd1b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.101+0000 7fd1b3fff640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:40:25.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.101+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b8004eb0 con 0x7fd1cc071e50 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 client.?.objecter ms_handle_connect 0x7fd1cc1457f0 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812958720 0 0) 0x7fd1cc1df280 con 0x7fd1cc1457f0 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc1df0f0 con 0x7fd1cc1457f0 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2478052039 0 0) 0x7fd1cc1df0f0 con 0x7fd1cc1457f0 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1cc1df280 con 0x7fd1cc1457f0 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd1bc0014b0 con 0x7fd1cc1457f0 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:25.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): dump: 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942606410 0 0) 0x7fd1cc0700a0 con 0x7fd1cc0a9d70 2024-03-29T10:40:25.093 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.102+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc1df0f0 con 0x7fd1cc0a9d70 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.103+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2118471529 0 0) 0x7fd1b8004eb0 con 0x7fd1cc071e50 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.103+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1cc0700a0 con 0x7fd1cc071e50 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.103+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd1c00017f0 con 0x7fd1cc071e50 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.103+0000 7fd1b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.103+0000 7fd1b3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.103+0000 7fd1b3fff640 10 monclient(hunting): dump: 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.094 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.103+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 827952520 0 0) 0x7fd1cc0700a0 con 0x7fd1cc071e50 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.103+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 >> v1:172.21.15.117:6790/0 conn(0x7fd1cc1457f0 legacy=0x7fd1cc1354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.103+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 >> v1:172.21.15.117:6789/0 conn(0x7fd1cc0a9d70 legacy=0x7fd1cc1de9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1b3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1b3fff640 1 monclient: found mon.b 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:25.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc06fc90 con 0x7fd1cc071e50 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1b3fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1b3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1d21d0640 5 monclient: authenticate success, global_id 4420 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1d21d0640 10 monclient: _renew_subs 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1d21d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc06f3d0 con 0x7fd1cc071e50 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1d21d0640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1d21d0640 10 monclient: _renew_subs 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1d21d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc0707a0 con 0x7fd1cc071e50 2024-03-29T10:40:25.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.104+0000 7fd1d21d0640 1 librados: init done 2024-03-29T10:40:25.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.106+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1c0001d00 con 0x7fd1cc071e50 2024-03-29T10:40:25.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.106+0000 7fd1b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:25.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.106+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd1c0003790 con 0x7fd1cc071e50 2024-03-29T10:40:25.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.106+0000 7fd1c9ffb640 4 set_mon_vals no callback set 2024-03-29T10:40:25.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.107+0000 7fd1d21d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.107+0000 7fd1d21d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.107+0000 7fd1d21d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.107+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ac005080 con 0x7fd1cc071e50 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.111+0000 7fd1b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.111+0000 7fd1b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.111+0000 7fd1b3fff640 10 monclient: dump: 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:25.099 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.111+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fd1c00197e0 con 0x7fd1cc071e50 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.112+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fd1c0083780 con 0x7fd1cc071e50 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.112+0000 7fd1b3fff640 10 client.4420.objecter ms_dispatch 0x7fd1cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.112+0000 7fd1b3fff640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.112+0000 7fd1b3fff640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.112+0000 7fd1b3fff640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.112+0000 7fd1cb7fe640 10 monclient: get_auth_request con 0x7fd1b806e0f0 auth_method 0 2024-03-29T10:40:25.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.112+0000 7fd1b3fff640 10 client.4420.objecter ms_handle_connect 0x7fd1b806e0f0 2024-03-29T10:40:25.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.117+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1c004ee90 con 0x7fd1cc071e50 2024-03-29T10:40:25.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.117+0000 7fd1b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:25.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.117+0000 7fd1b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:25.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.440+0000 7fd1d21d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:25.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.440+0000 7fd1d21d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:25.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.440+0000 7fd1d21d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:25.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.440+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1ac005330 con 0x7fd1cc071e50 2024-03-29T10:40:25.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.440+0000 7fd1b3fff640 1 -- 172.21.15.117:0/3795263482 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fd1c0050f90 con 0x7fd1cc071e50 2024-03-29T10:40:25.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.440+0000 7fd1b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:25.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.441+0000 7fd1b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:25.425 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:40:25.425 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:25.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.443+0000 7fd1d21d0640 10 librados: watch_flush enter 2024-03-29T10:40:25.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.443+0000 7fd1d21d0640 10 librados: watch_flush exit 2024-03-29T10:40:25.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.443+0000 7fd1d21d0640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-29T10:40:25.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.443+0000 7fd1d21d0640 10 client.4420.objecter successfully canceled tick 2024-03-29T10:40:25.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.445+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 >> v1:172.21.15.117:6816/376768171 conn(0x7fd1b806e0f0 legacy=0x7fd1b80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:25.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.445+0000 7fd1d21d0640 10 monclient: shutdown 2024-03-29T10:40:25.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.445+0000 7fd1d21d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:25.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.445+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 >> v1:172.21.15.136:6789/0 conn(0x7fd1cc071e50 legacy=0x7fd1cc1e6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:25.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.446+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 shutdown_connections 2024-03-29T10:40:25.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.446+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 >> 172.21.15.117:0/3795263482 conn(0x7fd1cc066a00 msgr2=0x7fd1cc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:25.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.446+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 shutdown_connections 2024-03-29T10:40:25.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.446+0000 7fd1d21d0640 1 -- 172.21.15.117:0/3795263482 wait complete. 2024-03-29T10:40:25.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:25.447+0000 7fd1d21d0640 1 librados: shutdown 2024-03-29T10:40:25.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:32.252 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:32.396 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.412+0000 7ff98e980640 10 monclient: get_monmap_and_config 2024-03-29T10:40:32.396 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.412+0000 7ff98e980640 10 monclient: build_initial_monmap 2024-03-29T10:40:32.396 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.412+0000 7ff98e980640 10 monclient: monmap: 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:40:32.413732+0000 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:40:32.413732+0000 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:40:32.397 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:32.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.413+0000 7ff98e980640 1 Processor -- start 2024-03-29T10:40:32.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.414+0000 7ff98e980640 1 -- start start 2024-03-29T10:40:32.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.414+0000 7ff98e980640 10 monclient: init 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 10 monclient(hunting): picked mon.noname-a con 0x7ff9880b2800 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 10 monclient(hunting): picked mon.noname-b con 0x7ff988142b20 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98d97e640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7ff9880b2800 0x7ff988142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:43740/0 (socket says 172.21.15.117:43740) 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 10 monclient(hunting): picked mon.noname-c con 0x7ff98813c040 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98d97e640 1 -- 172.21.15.117:0/1554243827 learned_addr learned my addr 172.21.15.117:0/1554243827 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 1 -- 172.21.15.117:0/1554243827 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9880aad10 con 0x7ff9880b2800 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 1 -- 172.21.15.117:0/1554243827 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff988141f30 con 0x7ff98813c040 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 1 -- 172.21.15.117:0/1554243827 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9880aaf60 con 0x7ff988142b20 2024-03-29T10:40:32.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.415+0000 7ff98e980640 10 monclient(hunting): authenticate will time out at 1153.677934s 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736705284 0 0) 0x7ff988141f30 con 0x7ff98813c040 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 10 monclient(hunting): _negotiate 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff974003520 con 0x7ff98813c040 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872595207 0 0) 0x7ff9880aad10 con 0x7ff9880b2800 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 10 monclient(hunting): _negotiate 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.416+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff988141f30 con 0x7ff9880b2800 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2186146025 0 0) 0x7ff974003520 con 0x7ff98813c040 2024-03-29T10:40:32.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 >> v1:172.21.15.117:6790/0 conn(0x7ff988142b20 legacy=0x7ff98813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 >> v1:172.21.15.117:6789/0 conn(0x7ff9880b2800 legacy=0x7ff988142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: _finish_hunting 0 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 1 monclient: found mon.noname-c 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880b3900 con 0x7ff98813c040 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: _finish_auth 0 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: _check_auth_tickets 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff980001a10 con 0x7ff98813c040 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: dump: 2024-03-29T10:40:32.405 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98e980640 5 monclient: authenticate success, global_id 4423 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98e980640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff980001c30 con 0x7ff98813c040 2024-03-29T10:40:32.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 1 -- 172.21.15.117:0/1554243827 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff980001fc0 con 0x7ff98813c040 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98e980640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98e980640 4 set_mon_vals no callback set 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.417+0000 7ff98c97c640 10 monclient: dump: 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:32.407 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.418+0000 7ff98e980640 10 monclient: shutdown 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.418+0000 7ff98e980640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.418+0000 7ff98e980640 1 -- 172.21.15.117:0/1554243827 >> v1:172.21.15.136:6789/0 conn(0x7ff98813c040 legacy=0x7ff98814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.418+0000 7ff98e980640 1 -- 172.21.15.117:0/1554243827 shutdown_connections 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.418+0000 7ff98e980640 1 -- 172.21.15.117:0/1554243827 >> 172.21.15.117:0/1554243827 conn(0x7ff98806d0e0 msgr2=0x7ff988066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.418+0000 7ff98e980640 1 -- 172.21.15.117:0/1554243827 shutdown_connections 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.418+0000 7ff98e980640 1 -- 172.21.15.117:0/1554243827 wait complete. 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.419+0000 7ff98e980640 10 monclient: build_initial_monmap 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.419+0000 7ff98e980640 10 monclient: monmap: 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:40:32.408 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:32.409 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:32.409 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:32.409 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:40:32.409 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:40:32.409 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:32.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.420+0000 7ff98e980640 1 librados: starting msgr at 2024-03-29T10:40:32.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.420+0000 7ff98e980640 1 librados: starting objecter 2024-03-29T10:40:32.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.420+0000 7ff98e980640 1 Processor -- start 2024-03-29T10:40:32.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.420+0000 7ff98e980640 1 -- start start 2024-03-29T10:40:32.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.420+0000 7ff98e980640 1 librados: setting wanted keys 2024-03-29T10:40:32.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.420+0000 7ff98e980640 1 librados: calling monclient init 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.420+0000 7ff98e980640 10 monclient: init 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 10 monclient(hunting): picked mon.noname-b con 0x7ff9880b2800 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 10 monclient(hunting): picked mon.noname-a con 0x7ff98813c040 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 10 monclient(hunting): picked mon.noname-c con 0x7ff988142b20 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff988143c40 con 0x7ff98813c040 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98d97e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff9880b2800 0x7ff98813c4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:47468/0 (socket says 172.21.15.117:47468) 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98d97e640 1 -- 172.21.15.117:0/3966632726 learned_addr learned my addr 172.21.15.117:0/3966632726 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff988143ea0 con 0x7ff9880b2800 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff97a7fc640 10 client.?.objecter ms_handle_connect 0x7ff98813c040 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff97a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff97a7fc640 10 client.?.objecter ms_handle_connect 0x7ff9880b2800 2024-03-29T10:40:32.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff97a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98813dba0 con 0x7ff988142b20 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.421+0000 7ff98e980640 10 monclient(hunting): authenticate will time out at 1153.684004s 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190170168 0 0) 0x7ff9880aaf60 con 0x7ff98813c040 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff96c004eb0 con 0x7ff98813c040 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 client.?.objecter ms_handle_connect 0x7ff988142b20 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16777339 0 0) 0x7ff988143ea0 con 0x7ff9880b2800 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:32.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9880aaf60 con 0x7ff9880b2800 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2037762901 0 0) 0x7ff9880aaf60 con 0x7ff9880b2800 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff988143ea0 con 0x7ff9880b2800 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff97c0019f0 con 0x7ff9880b2800 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.418 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626609888 0 0) 0x7ff98813dba0 con 0x7ff988142b20 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9880aaf60 con 0x7ff988142b20 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.422+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3483175850 0 0) 0x7ff96c004eb0 con 0x7ff98813c040 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff98813dba0 con 0x7ff98813c040 2024-03-29T10:40:32.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff984001620 con 0x7ff98813c040 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:32.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1204889838 0 0) 0x7ff988143ea0 con 0x7ff9880b2800 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 >> v1:172.21.15.117:6790/0 conn(0x7ff988142b20 legacy=0x7ff9881e6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 >> v1:172.21.15.117:6789/0 conn(0x7ff98813c040 legacy=0x7ff9881458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 1 monclient: found mon.b 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98813edc0 con 0x7ff9880b2800 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff97a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff98e980640 5 monclient: authenticate success, global_id 4426 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff98e980640 10 monclient: _renew_subs 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff98e980640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:32.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98813eaf0 con 0x7ff9880b2800 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff98e980640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff98e980640 10 monclient: _renew_subs 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff98e980640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98813cbf0 con 0x7ff9880b2800 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.423+0000 7ff98e980640 1 librados: init done 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.424+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff97c0047e0 con 0x7ff9880b2800 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.424+0000 7ff97a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.424+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff97c004b70 con 0x7ff9880b2800 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.424+0000 7ff97a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.424+0000 7ff97a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.424+0000 7ff97a7fc640 10 monclient: dump: 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:32.422 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.424+0000 7ff97bfff640 4 set_mon_vals no callback set 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.426+0000 7ff98e980640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.426+0000 7ff98e980640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.426+0000 7ff98e980640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.426+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff970005080 con 0x7ff9880b2800 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.426+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7ff97c004cd0 con 0x7ff9880b2800 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.427+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7ff97c084b20 con 0x7ff9880b2800 2024-03-29T10:40:32.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.427+0000 7ff97a7fc640 10 client.4426.objecter ms_dispatch 0x7ff988000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:32.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.427+0000 7ff97a7fc640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:32.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.427+0000 7ff97a7fc640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:32.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.427+0000 7ff97a7fc640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T10:40:32.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.427+0000 7ff98d17d640 10 monclient: get_auth_request con 0x7ff96c06e000 auth_method 0 2024-03-29T10:40:32.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.428+0000 7ff97a7fc640 10 client.4426.objecter ms_handle_connect 0x7ff96c06e000 2024-03-29T10:40:32.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.433+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff97c050400 con 0x7ff9880b2800 2024-03-29T10:40:32.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.433+0000 7ff97a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:32.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.433+0000 7ff97a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:32.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.850+0000 7ff98e980640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:32.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.850+0000 7ff98e980640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:32.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.850+0000 7ff98e980640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:32.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.850+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff970005330 con 0x7ff9880b2800 2024-03-29T10:40:32.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.851+0000 7ff97a7fc640 1 -- 172.21.15.117:0/3966632726 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7ff97c052330 con 0x7ff9880b2800 2024-03-29T10:40:32.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.851+0000 7ff97a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:32.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.851+0000 7ff97a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:32.836 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:40:32.836 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 10 librados: watch_flush enter 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 10 librados: watch_flush exit 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 10 client.4426.objecter successfully canceled tick 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 >> v1:172.21.15.117:6816/376768171 conn(0x7ff96c06e000 legacy=0x7ff96c0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 10 monclient: shutdown 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 >> v1:172.21.15.136:6789/0 conn(0x7ff9880b2800 legacy=0x7ff98813c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 shutdown_connections 2024-03-29T10:40:32.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 >> 172.21.15.117:0/3966632726 conn(0x7ff98806d0e0 msgr2=0x7ff988066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:32.841 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 shutdown_connections 2024-03-29T10:40:32.841 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 1 -- 172.21.15.117:0/3966632726 wait complete. 2024-03-29T10:40:32.841 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:32.855+0000 7ff98e980640 1 librados: shutdown 2024-03-29T10:40:32.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:39.669 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:39.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.832+0000 7f1a75451640 10 monclient: get_monmap_and_config 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.832+0000 7f1a75451640 10 monclient: build_initial_monmap 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.832+0000 7f1a75451640 10 monclient: monmap: 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:40:39.832953+0000 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:40:39.832953+0000 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:40:39.816 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:39.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.833+0000 7f1a75451640 1 Processor -- start 2024-03-29T10:40:39.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.833+0000 7f1a75451640 1 -- start start 2024-03-29T10:40:39.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.833+0000 7f1a75451640 10 monclient: init 2024-03-29T10:40:39.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:39.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:39.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 10 monclient(hunting): picked mon.noname-a con 0x7f1a68006eb0 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:39.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 10 monclient(hunting): picked mon.noname-b con 0x7f1a68038f10 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:39.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 10 monclient(hunting): picked mon.noname-c con 0x7f1a68036690 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:39.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a68033c90 con 0x7f1a68006eb0 2024-03-29T10:40:39.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a680389f0 con 0x7f1a68036690 2024-03-29T10:40:39.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a68030890 con 0x7f1a68038f10 2024-03-29T10:40:39.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:39.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.834+0000 7f1a75451640 10 monclient(hunting): authenticate will time out at 1161.096212s 2024-03-29T10:40:39.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.840+0000 7f1a6f7fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f1a68038f10 0x7f1a68039330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:36329/0 (socket says 172.21.15.117:36329) 2024-03-29T10:40:39.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.840+0000 7f1a6f7fe640 1 -- 172.21.15.117:0/3729914672 learned_addr learned my addr 172.21.15.117:0/3729914672 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:39.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516888306 0 0) 0x7f1a68030890 con 0x7f1a68038f10 2024-03-29T10:40:39.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a60003520 con 0x7f1a68038f10 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032051038 0 0) 0x7f1a68033c90 con 0x7f1a68006eb0 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a68030890 con 0x7f1a68006eb0 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845120125 0 0) 0x7f1a68030890 con 0x7f1a68006eb0 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 >> v1:172.21.15.117:6790/0 conn(0x7f1a68038f10 legacy=0x7f1a68039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.841+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 >> v1:172.21.15.136:6789/0 conn(0x7f1a68036690 legacy=0x7f1a68036b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.842+0000 7f1a6effd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.842+0000 7f1a6effd640 1 monclient: found mon.noname-a 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.842+0000 7f1a6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.842+0000 7f1a6effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.842+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a680cbc40 con 0x7f1a68006eb0 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.842+0000 7f1a6effd640 10 monclient: _finish_auth 0 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.842+0000 7f1a6effd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.842+0000 7f1a6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a75451640 5 monclient: authenticate success, global_id 4443 2024-03-29T10:40:39.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a75451640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1a64001690 con 0x7f1a68006eb0 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 10 monclient: dump: 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:39.831 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a75451640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a640019a0 con 0x7f1a68006eb0 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a75451640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a75451640 4 set_mon_vals no callback set 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 1 -- 172.21.15.117:0/3729914672 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1a64001d70 con 0x7f1a68006eb0 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.843+0000 7f1a6effd640 10 monclient: dump: 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:39.832 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.844+0000 7f1a75451640 10 monclient: shutdown 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.844+0000 7f1a75451640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.844+0000 7f1a75451640 1 -- 172.21.15.117:0/3729914672 >> v1:172.21.15.117:6789/0 conn(0x7f1a68006eb0 legacy=0x7f1a68031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.844+0000 7f1a75451640 1 -- 172.21.15.117:0/3729914672 shutdown_connections 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.844+0000 7f1a75451640 1 -- 172.21.15.117:0/3729914672 >> 172.21.15.117:0/3729914672 conn(0x7f1a68005c20 msgr2=0x7f1a68006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.844+0000 7f1a75451640 1 -- 172.21.15.117:0/3729914672 shutdown_connections 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.844+0000 7f1a75451640 1 -- 172.21.15.117:0/3729914672 wait complete. 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.845+0000 7f1a75451640 10 monclient: build_initial_monmap 2024-03-29T10:40:39.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.845+0000 7f1a75451640 10 monclient: monmap: 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.845+0000 7f1a75451640 1 librados: starting msgr at 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.845+0000 7f1a75451640 1 librados: starting objecter 2024-03-29T10:40:39.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.845+0000 7f1a75451640 1 Processor -- start 2024-03-29T10:40:39.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.846+0000 7f1a75451640 1 -- start start 2024-03-29T10:40:39.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.846+0000 7f1a75451640 1 librados: setting wanted keys 2024-03-29T10:40:39.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.846+0000 7f1a75451640 1 librados: calling monclient init 2024-03-29T10:40:39.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.846+0000 7f1a75451640 10 monclient: init 2024-03-29T10:40:39.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 10 monclient(hunting): picked mon.noname-c con 0x7f1a68006eb0 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 10 monclient(hunting): picked mon.noname-b con 0x7f1a68036690 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 10 monclient(hunting): picked mon.noname-a con 0x7f1a68038f10 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6800b680 con 0x7f1a68038f10 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a680d0000 con 0x7f1a68036690 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6800b8d0 con 0x7f1a68006eb0 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.847+0000 7f1a75451640 10 monclient(hunting): authenticate will time out at 1161.109451s 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.851+0000 7f1a6f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1a68036690 0x7f1a680097b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:45853/0 (socket says 172.21.15.117:45853) 2024-03-29T10:40:39.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.851+0000 7f1a6f7fe640 1 -- 172.21.15.117:0/921867099 learned_addr learned my addr 172.21.15.117:0/921867099 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:39.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.854+0000 7f1a6cff9640 10 client.?.objecter ms_handle_connect 0x7f1a68036690 2024-03-29T10:40:39.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.854+0000 7f1a6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:39.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.854+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079451224 0 0) 0x7f1a680d0000 con 0x7f1a68036690 2024-03-29T10:40:39.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.854+0000 7f1a6cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:39.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.854+0000 7f1a6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:39.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.854+0000 7f1a6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:39.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.854+0000 7f1a6cff9640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:40:39.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.854+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a54004eb0 con 0x7f1a68036690 2024-03-29T10:40:39.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.855+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 920893279 0 0) 0x7f1a54004eb0 con 0x7f1a68036690 2024-03-29T10:40:39.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.855+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a680d0000 con 0x7f1a68036690 2024-03-29T10:40:39.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.855+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1a7004fdd0 con 0x7f1a68036690 2024-03-29T10:40:39.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.855+0000 7f1a6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:39.839 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.855+0000 7f1a6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.855+0000 7f1a6cff9640 10 monclient(hunting): dump: 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.855+0000 7f1a6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:39.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.856+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2443814479 0 0) 0x7f1a680d0000 con 0x7f1a68036690 2024-03-29T10:40:39.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 >> v1:172.21.15.117:6790/0 conn(0x7f1a68006eb0 legacy=0x7f1a6800c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 >> v1:172.21.15.117:6789/0 conn(0x7f1a68038f10 legacy=0x7f1a68009ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 1 monclient: found mon.b 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6800bbb0 con 0x7f1a68036690 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 10 monclient: _finish_auth 0 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a75451640 5 monclient: authenticate success, global_id 4429 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a75451640 10 monclient: _renew_subs 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a75451640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:39.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6800d610 con 0x7f1a68036690 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a75451640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a75451640 10 monclient: _renew_subs 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a75451640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6800c520 con 0x7f1a68036690 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.857+0000 7f1a75451640 1 librados: init done 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.858+0000 7f1a6cff9640 10 client.4429.objecter ms_handle_connect 0x7f1a68006eb0 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.858+0000 7f1a6cff9640 10 client.4429.objecter resend_mon_ops 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.858+0000 7f1a75451640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.858+0000 7f1a75451640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.858+0000 7f1a75451640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.858+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a68007b90 con 0x7f1a68036690 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.859+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a7004f400 con 0x7f1a68036690 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.859+0000 7f1a6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.859+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1a70075f10 con 0x7f1a68036690 2024-03-29T10:40:39.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.859+0000 7f1a6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.859+0000 7f1a6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.859+0000 7f1a6cff9640 10 monclient: dump: 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:39.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.859+0000 7f1a6dffb640 4 set_mon_vals no callback set 2024-03-29T10:40:39.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.861+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f1a70067bf0 con 0x7f1a68036690 2024-03-29T10:40:39.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.861+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f1a700ff5b0 con 0x7f1a68036690 2024-03-29T10:40:39.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.861+0000 7f1a6cff9640 10 client.4429.objecter ms_dispatch 0x7f1a70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:39.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.861+0000 7f1a6cff9640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:39.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.861+0000 7f1a6cff9640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:39.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.861+0000 7f1a6cff9640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-29T10:40:39.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.863+0000 7f1a6ffff640 10 monclient: get_auth_request con 0x7f1a5406dad0 auth_method 0 2024-03-29T10:40:39.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.865+0000 7f1a6cff9640 10 client.4429.objecter ms_handle_connect 0x7f1a5406dad0 2024-03-29T10:40:39.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.865+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a700caef0 con 0x7f1a68036690 2024-03-29T10:40:39.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.865+0000 7f1a6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:39.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:39.865+0000 7f1a6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:40.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.229+0000 7f1a75451640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:40.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.229+0000 7f1a75451640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:40.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.229+0000 7f1a75451640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:40:40.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.229+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a68004cd0 con 0x7f1a68036690 2024-03-29T10:40:40.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.230+0000 7f1a6cff9640 1 -- 172.21.15.117:0/921867099 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f1a700cce20 con 0x7f1a68036690 2024-03-29T10:40:40.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.230+0000 7f1a6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:40.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.230+0000 7f1a6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:40.214 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:40:40.214 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:40.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.233+0000 7f1a75451640 10 librados: watch_flush enter 2024-03-29T10:40:40.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.233+0000 7f1a75451640 10 librados: watch_flush exit 2024-03-29T10:40:40.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.233+0000 7f1a75451640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-29T10:40:40.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.233+0000 7f1a75451640 10 client.4429.objecter successfully canceled tick 2024-03-29T10:40:40.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.233+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 >> v1:172.21.15.117:6816/376768171 conn(0x7f1a5406dad0 legacy=0x7f1a5406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:40.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.234+0000 7f1a75451640 10 monclient: shutdown 2024-03-29T10:40:40.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.234+0000 7f1a75451640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:40.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.234+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 >> v1:172.21.15.136:6789/0 conn(0x7f1a68036690 legacy=0x7f1a680097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:40.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.234+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 shutdown_connections 2024-03-29T10:40:40.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.234+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 >> 172.21.15.117:0/921867099 conn(0x7f1a68005c20 msgr2=0x7f1a680043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:40.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.234+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 shutdown_connections 2024-03-29T10:40:40.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.234+0000 7f1a75451640 1 -- 172.21.15.117:0/921867099 wait complete. 2024-03-29T10:40:40.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:40.234+0000 7f1a75451640 1 librados: shutdown 2024-03-29T10:40:40.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:47.042 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.215+0000 7f11914a8640 10 monclient: get_monmap_and_config 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.215+0000 7f11914a8640 10 monclient: build_initial_monmap 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.215+0000 7f11914a8640 10 monclient: monmap: 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:40:47.216218+0000 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:40:47.216218+0000 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:40:47.199 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:40:47.200 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.216+0000 7f11914a8640 1 Processor -- start 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.216+0000 7f11914a8640 1 -- start start 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.216+0000 7f11914a8640 10 monclient: init 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 10 monclient(hunting): picked mon.noname-c con 0x7f118c1457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 10 monclient(hunting): picked mon.noname-a con 0x7f118c13f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 10 monclient(hunting): picked mon.noname-b con 0x7f118c0a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0aaef0 con 0x7f118c13f650 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c145bd0 con 0x7f118c1457f0 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c145eb0 con 0x7f118c0a9d70 2024-03-29T10:40:47.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:47.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.217+0000 7f11914a8640 10 monclient(hunting): authenticate will time out at 1168.479677s 2024-03-29T10:40:47.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.220+0000 7f118affd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f118c1457f0 0x7f118c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:39149/0 (socket says 172.21.15.117:39149) 2024-03-29T10:40:47.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.220+0000 7f118affd640 1 -- 172.21.15.117:0/3195840374 learned_addr learned my addr 172.21.15.117:0/3195840374 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:47.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506551433 0 0) 0x7f118c145bd0 con 0x7f118c1457f0 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1174003520 con 0x7f118c1457f0 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673406507 0 0) 0x7f118c145eb0 con 0x7f118c0a9d70 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118c145bd0 con 0x7f118c0a9d70 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421782246 0 0) 0x7f118c0aaef0 con 0x7f118c13f650 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:47.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.221+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118c145eb0 con 0x7f118c13f650 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4220253913 0 0) 0x7f118c145bd0 con 0x7f118c0a9d70 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 >> v1:172.21.15.136:6789/0 conn(0x7f118c1457f0 legacy=0x7f118c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 >> v1:172.21.15.117:6789/0 conn(0x7f118c13f650 legacy=0x7f118c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 1 monclient: found mon.noname-b 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c146340 con 0x7f118c0a9d70 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:47.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f11914a8640 5 monclient: authenticate success, global_id 4493 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f11914a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1180001670 con 0x7f118c0a9d70 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f1189ffb640 10 monclient: dump: 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:47.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.222+0000 7f11914a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.225+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11800017d0 con 0x7f118c0a9d70 2024-03-29T10:40:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.225+0000 7f1189ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.225+0000 7f11914a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:47.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.225+0000 7f11914a8640 4 set_mon_vals no callback set 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f1189ffb640 1 -- 172.21.15.117:0/3195840374 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1180001be0 con 0x7f118c0a9d70 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f1189ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f1189ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f1189ffb640 10 monclient: dump: 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:47.213 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f11914a8640 10 monclient: shutdown 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f11914a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f11914a8640 1 -- 172.21.15.117:0/3195840374 >> v1:172.21.15.117:6790/0 conn(0x7f118c0a9d70 legacy=0x7f118c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f11914a8640 1 -- 172.21.15.117:0/3195840374 shutdown_connections 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f11914a8640 1 -- 172.21.15.117:0/3195840374 >> 172.21.15.117:0/3195840374 conn(0x7f118c066a00 msgr2=0x7f118c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f11914a8640 1 -- 172.21.15.117:0/3195840374 shutdown_connections 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.226+0000 7f11914a8640 1 -- 172.21.15.117:0/3195840374 wait complete. 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.227+0000 7f11914a8640 10 monclient: build_initial_monmap 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.227+0000 7f11914a8640 10 monclient: monmap: 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:47.214 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.228+0000 7f11914a8640 1 librados: starting msgr at 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.228+0000 7f11914a8640 1 librados: starting objecter 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.228+0000 7f11914a8640 1 Processor -- start 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.229+0000 7f11914a8640 1 -- start start 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.229+0000 7f11914a8640 1 librados: setting wanted keys 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.229+0000 7f11914a8640 1 librados: calling monclient init 2024-03-29T10:40:47.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.229+0000 7f11914a8640 10 monclient: init 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.230+0000 7f11914a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.230+0000 7f11914a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.230+0000 7f11914a8640 10 monclient(hunting): picked mon.noname-c con 0x7f118c0a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f118affd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f118c0a9d70 0x7f118c1d7470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58315/0 (socket says 172.21.15.117:58315) 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f11914a8640 10 monclient(hunting): picked mon.noname-b con 0x7f118c13f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f118affd640 1 -- 172.21.15.117:0/2461200209 learned_addr learned my addr 172.21.15.117:0/2461200209 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f11914a8640 10 monclient(hunting): picked mon.noname-a con 0x7f118c1457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f11737fe640 10 client.?.objecter ms_handle_connect 0x7f118c0a9d70 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f11737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c13a690 con 0x7f118c1457f0 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c13a970 con 0x7f118c13f650 2024-03-29T10:40:47.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c135890 con 0x7f118c0a9d70 2024-03-29T10:40:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.233+0000 7f11914a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.234+0000 7f11914a8640 10 monclient(hunting): authenticate will time out at 1168.496105s 2024-03-29T10:40:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.234+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117020328 0 0) 0x7f118c135890 con 0x7f118c0a9d70 2024-03-29T10:40:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.234+0000 7f11737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.234+0000 7f11737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.234+0000 7f11737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.234+0000 7f11737fe640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:40:47.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.234+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1178004eb0 con 0x7f118c0a9d70 2024-03-29T10:40:47.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.234+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2475900318 0 0) 0x7f1178004eb0 con 0x7f118c0a9d70 2024-03-29T10:40:47.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.234+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f118c135890 con 0x7f118c0a9d70 2024-03-29T10:40:47.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1184001bb0 con 0x7f118c0a9d70 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 10 monclient(hunting): dump: 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:47.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 783015924 0 0) 0x7f118c135890 con 0x7f118c0a9d70 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 >> v1:172.21.15.136:6789/0 conn(0x7f118c13f650 legacy=0x7f118c134a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 >> v1:172.21.15.117:6789/0 conn(0x7f118c1457f0 legacy=0x7f118c135180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 1 monclient: found mon.c 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c138a90 con 0x7f118c0a9d70 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.235+0000 7f11737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.236+0000 7f11914a8640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.236+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1184004b60 con 0x7f118c0a9d70 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.236+0000 7f11737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:47.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.236+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1184001bb0 con 0x7f118c0a9d70 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.236+0000 7f11737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.236+0000 7f11737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.236+0000 7f11737fe640 10 monclient: dump: 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:47.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.236+0000 7f11897fa640 4 set_mon_vals no callback set 2024-03-29T10:40:47.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.237+0000 7f11914a8640 10 monclient: _renew_subs 2024-03-29T10:40:47.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.237+0000 7f11914a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:47.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.237+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c1d7b80 con 0x7f118c0a9d70 2024-03-29T10:40:47.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.237+0000 7f11914a8640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:47.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.237+0000 7f11914a8640 10 monclient: _renew_subs 2024-03-29T10:40:47.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.237+0000 7f11914a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:47.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.237+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c13b8d0 con 0x7f118c0a9d70 2024-03-29T10:40:47.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.237+0000 7f11914a8640 1 librados: init done 2024-03-29T10:40:47.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.239+0000 7f11914a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:47.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.239+0000 7f11914a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:47.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.239+0000 7f11914a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:47.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.239+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c13b260 con 0x7f118c0a9d70 2024-03-29T10:40:47.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.239+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f1184004e50 con 0x7f118c0a9d70 2024-03-29T10:40:47.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.240+0000 7f118a7fc640 10 monclient: get_auth_request con 0x7f117806de60 auth_method 0 2024-03-29T10:40:47.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.240+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f1184054a10 con 0x7f118c0a9d70 2024-03-29T10:40:47.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.240+0000 7f11737fe640 10 client.4496.objecter ms_dispatch 0x7f118c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:47.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.240+0000 7f11737fe640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:47.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.240+0000 7f11737fe640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:47.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.240+0000 7f11737fe640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T10:40:47.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.241+0000 7f11737fe640 10 client.4496.objecter ms_handle_connect 0x7f117806de60 2024-03-29T10:40:47.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.246+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1184050cb0 con 0x7f118c0a9d70 2024-03-29T10:40:47.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.246+0000 7f11737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:47.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.246+0000 7f11737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:47.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.566+0000 7f11914a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:47.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.567+0000 7f11914a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:47.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.567+0000 7f11914a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:47.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.567+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f118c13b070 con 0x7f118c0a9d70 2024-03-29T10:40:47.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.570+0000 7f11737fe640 1 -- 172.21.15.117:0/2461200209 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f1184050cb0 con 0x7f118c0a9d70 2024-03-29T10:40:47.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.570+0000 7f11737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:47.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.570+0000 7f11737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:47.555 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:40:47.555 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:47.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.573+0000 7f11914a8640 10 librados: watch_flush enter 2024-03-29T10:40:47.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 10 librados: watch_flush exit 2024-03-29T10:40:47.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T10:40:47.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 10 client.4496.objecter successfully canceled tick 2024-03-29T10:40:47.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 >> v1:172.21.15.117:6816/376768171 conn(0x7f117806de60 legacy=0x7f1178070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:47.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 10 monclient: shutdown 2024-03-29T10:40:47.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:47.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 >> v1:172.21.15.117:6790/0 conn(0x7f118c0a9d70 legacy=0x7f118c1d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:47.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 shutdown_connections 2024-03-29T10:40:47.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 >> 172.21.15.117:0/2461200209 conn(0x7f118c066a00 msgr2=0x7f118c0b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:47.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.574+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 shutdown_connections 2024-03-29T10:40:47.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.575+0000 7f11914a8640 1 -- 172.21.15.117:0/2461200209 wait complete. 2024-03-29T10:40:47.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:47.575+0000 7f11914a8640 1 librados: shutdown 2024-03-29T10:40:47.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:54.385 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:54.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.562+0000 7ff1980f4640 10 monclient: get_monmap_and_config 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.562+0000 7ff1980f4640 10 monclient: build_initial_monmap 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.562+0000 7ff1980f4640 10 monclient: monmap: 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:40:54.564040+0000 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:40:54.564040+0000 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:40:54.547 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:54.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.563+0000 7ff1980f4640 1 Processor -- start 2024-03-29T10:40:54.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.563+0000 7ff1980f4640 1 -- start start 2024-03-29T10:40:54.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.563+0000 7ff1980f4640 10 monclient: init 2024-03-29T10:40:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff1980f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff1980f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff1980f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff1900b0860 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff1980f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff19013ef30 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff1980f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff1900af8e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:54.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff19568f640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7ff19013ef30 0x7ff1900af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:51758/0 (socket says 172.21.15.117:51758) 2024-03-29T10:40:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff19568f640 1 -- 172.21.15.117:0/3235326611 learned_addr learned my addr 172.21.15.117:0/3235326611 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff1980f4640 1 -- 172.21.15.117:0/3235326611 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900b0c40 con 0x7ff19013ef30 2024-03-29T10:40:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff1980f4640 1 -- 172.21.15.117:0/3235326611 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900aed10 con 0x7ff1900b0860 2024-03-29T10:40:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.564+0000 7ff1980f4640 1 -- 172.21.15.117:0/3235326611 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900afd50 con 0x7ff1900af8e0 2024-03-29T10:40:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff1980f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:54.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff1980f4640 10 monclient(hunting): authenticate will time out at 1175.828230s 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665861599 0 0) 0x7ff1900aed10 con 0x7ff1900b0860 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): _negotiate 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff17c003db0 con 0x7ff1900b0860 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044072007 0 0) 0x7ff1900b0c40 con 0x7ff19013ef30 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): _negotiate 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1900aed10 con 0x7ff19013ef30 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885988772 0 0) 0x7ff1900afd50 con 0x7ff1900af8e0 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): _negotiate 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.565+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1900b0c40 con 0x7ff1900af8e0 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694755132 0 0) 0x7ff17c003db0 con 0x7ff1900b0860 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 >> v1:172.21.15.117:6790/0 conn(0x7ff1900af8e0 legacy=0x7ff190149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 >> v1:172.21.15.117:6789/0 conn(0x7ff19013ef30 legacy=0x7ff1900af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 10 monclient: _finish_hunting 0 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 1 monclient: found mon.noname-c 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:40:54.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1900ab1b0 con 0x7ff1900b0860 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 10 monclient: _finish_auth 0 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 10 monclient: _check_auth_tickets 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff18c001690 con 0x7ff1900b0860 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff194e8e640 10 monclient: dump: 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:54.555 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:54.556 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:54.556 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:54.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff1980f4640 5 monclient: authenticate success, global_id 4435 2024-03-29T10:40:54.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.566+0000 7ff1980f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.569+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff18c0017f0 con 0x7ff1900b0860 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.569+0000 7ff194e8e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.569+0000 7ff194e8e640 1 -- 172.21.15.117:0/3235326611 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff18c001c00 con 0x7ff1900b0860 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.569+0000 7ff1980f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.569+0000 7ff1980f4640 4 set_mon_vals no callback set 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.570+0000 7ff194e8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.570+0000 7ff194e8e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.570+0000 7ff194e8e640 10 monclient: dump: 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.570+0000 7ff1980f4640 10 monclient: shutdown 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.570+0000 7ff1980f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.570+0000 7ff1980f4640 1 -- 172.21.15.117:0/3235326611 >> v1:172.21.15.136:6789/0 conn(0x7ff1900b0860 legacy=0x7ff19013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.570+0000 7ff1980f4640 1 -- 172.21.15.117:0/3235326611 shutdown_connections 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.570+0000 7ff1980f4640 1 -- 172.21.15.117:0/3235326611 >> 172.21.15.117:0/3235326611 conn(0x7ff19006b660 msgr2=0x7ff19006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.570+0000 7ff1980f4640 1 -- 172.21.15.117:0/3235326611 shutdown_connections 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.571+0000 7ff1980f4640 1 -- 172.21.15.117:0/3235326611 wait complete. 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.572+0000 7ff1980f4640 10 monclient: build_initial_monmap 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.572+0000 7ff1980f4640 10 monclient: monmap: 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:54.559 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.572+0000 7ff1980f4640 1 librados: starting msgr at 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.572+0000 7ff1980f4640 1 librados: starting objecter 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.572+0000 7ff1980f4640 1 Processor -- start 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.572+0000 7ff1980f4640 1 -- start start 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.572+0000 7ff1980f4640 1 librados: setting wanted keys 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.572+0000 7ff1980f4640 1 librados: calling monclient init 2024-03-29T10:40:54.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.572+0000 7ff1980f4640 10 monclient: init 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff1900af8e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff1900b0860 addr v1:172.21.15.117:6790/0 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff19013ef30 addr v1:172.21.15.136:6789/0 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900b3da0 con 0x7ff1900af8e0 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900b3790 con 0x7ff19013ef30 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900b3920 con 0x7ff1900b0860 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1980f4640 10 monclient(hunting): authenticate will time out at 1175.836592s 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff19568f640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ff1900b0860 0x7ff190133690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:37958/0 (socket says 172.21.15.117:37958) 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff19568f640 1 -- 172.21.15.117:0/2241981458 learned_addr learned my addr 172.21.15.117:0/2241981458 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:40:54.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1867fc640 10 client.?.objecter ms_handle_connect 0x7ff1900b0860 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1867fc640 10 client.?.objecter ms_handle_connect 0x7ff1900af8e0 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.573+0000 7ff1867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195947925 0 0) 0x7ff1900b3920 con 0x7ff1900b0860 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff174004eb0 con 0x7ff1900b0860 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913713977 0 0) 0x7ff1900b3da0 con 0x7ff1900af8e0 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:54.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1900b3920 con 0x7ff1900af8e0 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 client.?.objecter ms_handle_connect 0x7ff19013ef30 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 781516474 0 0) 0x7ff174004eb0 con 0x7ff1900b0860 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1900b3da0 con 0x7ff1900b0860 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff1800018b0 con 0x7ff1900b0860 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): dump: 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.565 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 455931997 0 0) 0x7ff1900b3920 con 0x7ff1900af8e0 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff174004eb0 con 0x7ff1900af8e0 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff18c001bc0 con 0x7ff1900af8e0 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.574+0000 7ff1867fc640 10 monclient(hunting): dump: 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.566 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3441587026 0 0) 0x7ff1900b3da0 con 0x7ff1900b0860 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 >> v1:172.21.15.136:6789/0 conn(0x7ff19013ef30 legacy=0x7ff1901d6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 >> v1:172.21.15.117:6789/0 conn(0x7ff1900af8e0 legacy=0x7ff1900b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 1 monclient: found mon.c 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:54.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1900b3ab0 con 0x7ff1900b0860 2024-03-29T10:40:54.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:54.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:54.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:54.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1980f4640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:40:54.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1980f4640 10 monclient: _renew_subs 2024-03-29T10:40:54.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1980f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:54.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1901c7ae0 con 0x7ff1900b0860 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1980f4640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1980f4640 10 monclient: _renew_subs 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1980f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff190133da0 con 0x7ff1900b0860 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1980f4640 1 librados: init done 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1800047e0 con 0x7ff1900b0860 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff180004af0 con 0x7ff1900b0860 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff1867fc640 10 monclient: dump: 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:40:54.569 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.570 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:40:54.570 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.570 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:40:54.570 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:40:54.570 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:40:54.570 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:40:54.570 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:40:54.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.575+0000 7ff187fff640 4 set_mon_vals no callback set 2024-03-29T10:40:54.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.577+0000 7ff1980f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:54.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.578+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7ff18001aac0 con 0x7ff1900b0860 2024-03-29T10:40:54.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.577+0000 7ff1980f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:54.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.579+0000 7ff1980f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:54.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.579+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff178005080 con 0x7ff1900b0860 2024-03-29T10:40:54.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.579+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7ff180084a50 con 0x7ff1900b0860 2024-03-29T10:40:54.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.579+0000 7ff1867fc640 10 client.4502.objecter ms_dispatch 0x7ff190000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:54.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.579+0000 7ff1867fc640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:54.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.579+0000 7ff1867fc640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:54.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.579+0000 7ff1867fc640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T10:40:54.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.579+0000 7ff195e90640 10 monclient: get_auth_request con 0x7ff17406de90 auth_method 0 2024-03-29T10:40:54.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.580+0000 7ff1867fc640 10 client.4502.objecter ms_handle_connect 0x7ff17406de90 2024-03-29T10:40:54.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.587+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff180050160 con 0x7ff1900b0860 2024-03-29T10:40:54.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.587+0000 7ff1867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:54.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.587+0000 7ff1867fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:54.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.991+0000 7ff1980f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:54.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.991+0000 7ff1980f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:54.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.991+0000 7ff1980f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:40:54.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.991+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff178005330 con 0x7ff1900b0860 2024-03-29T10:40:54.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.992+0000 7ff1867fc640 1 -- 172.21.15.117:0/2241981458 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7ff180052260 con 0x7ff1900b0860 2024-03-29T10:40:54.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.992+0000 7ff1867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:54.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.992+0000 7ff1867fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:54.977 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:40:54.977 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:54.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.995+0000 7ff1980f4640 10 librados: watch_flush enter 2024-03-29T10:40:54.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.995+0000 7ff1980f4640 10 librados: watch_flush exit 2024-03-29T10:40:54.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.995+0000 7ff1980f4640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T10:40:54.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.995+0000 7ff1980f4640 10 client.4502.objecter successfully canceled tick 2024-03-29T10:40:54.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.996+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 >> v1:172.21.15.117:6816/376768171 conn(0x7ff17406de90 legacy=0x7ff174070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.996+0000 7ff1980f4640 10 monclient: shutdown 2024-03-29T10:40:54.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.996+0000 7ff1980f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:54.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.996+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 >> v1:172.21.15.117:6790/0 conn(0x7ff1900b0860 legacy=0x7ff190133690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.996+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 shutdown_connections 2024-03-29T10:40:54.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.996+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 >> 172.21.15.117:0/2241981458 conn(0x7ff19006b660 msgr2=0x7ff1900ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:54.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.996+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 shutdown_connections 2024-03-29T10:40:54.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.996+0000 7ff1980f4640 1 -- 172.21.15.117:0/2241981458 wait complete. 2024-03-29T10:40:54.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:40:54.996+0000 7ff1980f4640 1 librados: shutdown 2024-03-29T10:40:54.996 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:01.803 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:01.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.973+0000 7f0e750ab640 10 monclient: get_monmap_and_config 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.973+0000 7f0e750ab640 10 monclient: build_initial_monmap 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.973+0000 7f0e750ab640 10 monclient: monmap: 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:41:01.974688+0000 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:41:01.974688+0000 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:41:01.958 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:01.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 1 Processor -- start 2024-03-29T10:41:01.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 1 -- start start 2024-03-29T10:41:01.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 10 monclient: init 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 10 monclient(hunting): picked mon.noname-b con 0x7f0e68006a70 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 10 monclient(hunting): picked mon.noname-a con 0x7f0e6802b100 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 10 monclient(hunting): picked mon.noname-c con 0x7f0e68029c30 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6802a270 con 0x7f0e6802b100 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6802c440 con 0x7f0e68029c30 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e680c9440 con 0x7f0e68006a70 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.975+0000 7f0e750ab640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.976+0000 7f0e750ab640 10 monclient(hunting): authenticate will time out at 1183.238104s 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.976+0000 7f0e748aa640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0e68029c30 0x7f0e680d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:56438/0 (socket says 172.21.15.117:56438) 2024-03-29T10:41:01.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.976+0000 7f0e748aa640 1 -- 172.21.15.117:0/2506825547 learned_addr learned my addr 172.21.15.117:0/2506825547 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:01.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.978+0000 7f0e6effd640 1 -- 172.21.15.117:0/2506825547 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683849983 0 0) 0x7f0e6802c440 con 0x7f0e68029c30 2024-03-29T10:41:01.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.978+0000 7f0e6effd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:01.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.978+0000 7f0e6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:01.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.978+0000 7f0e6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:01.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.978+0000 7f0e6effd640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:41:01.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.978+0000 7f0e6effd640 1 -- 172.21.15.117:0/2506825547 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e60003520 con 0x7f0e68029c30 2024-03-29T10:41:01.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.979+0000 7f0e6effd640 1 -- 172.21.15.117:0/2506825547 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1568474476 0 0) 0x7f0e60003520 con 0x7f0e68029c30 2024-03-29T10:41:01.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.979+0000 7f0e6effd640 1 -- 172.21.15.117:0/2506825547 >> v1:172.21.15.117:6790/0 conn(0x7f0e68006a70 legacy=0x7f0e680c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:01.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.979+0000 7f0e6effd640 1 -- 172.21.15.117:0/2506825547 >> v1:172.21.15.117:6789/0 conn(0x7f0e6802b100 legacy=0x7f0e680cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:01.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:01.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 1 monclient: found mon.noname-c 2024-03-29T10:41:01.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 1 -- 172.21.15.117:0/2506825547 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e680c95d0 con 0x7f0e68029c30 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: _finish_auth 0 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e750ab640 5 monclient: authenticate success, global_id 4441 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e750ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 1 -- 172.21.15.117:0/2506825547 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0e64001690 con 0x7f0e68029c30 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: dump: 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.967 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e750ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 1 -- 172.21.15.117:0/2506825547 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e64001860 con 0x7f0e68029c30 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e750ab640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e750ab640 4 set_mon_vals no callback set 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 1 -- 172.21.15.117:0/2506825547 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0e64001cb0 con 0x7f0e68029c30 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:01.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.982+0000 7f0e6effd640 10 monclient: dump: 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:01.969 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:01.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.983+0000 7f0e750ab640 10 monclient: shutdown 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.983+0000 7f0e750ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.983+0000 7f0e750ab640 1 -- 172.21.15.117:0/2506825547 >> v1:172.21.15.136:6789/0 conn(0x7f0e68029c30 legacy=0x7f0e680d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.984+0000 7f0e750ab640 1 -- 172.21.15.117:0/2506825547 shutdown_connections 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.984+0000 7f0e750ab640 1 -- 172.21.15.117:0/2506825547 >> 172.21.15.117:0/2506825547 conn(0x7f0e680058f0 msgr2=0x7f0e68005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.984+0000 7f0e750ab640 1 -- 172.21.15.117:0/2506825547 shutdown_connections 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.984+0000 7f0e750ab640 1 -- 172.21.15.117:0/2506825547 wait complete. 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.985+0000 7f0e750ab640 10 monclient: build_initial_monmap 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.985+0000 7f0e750ab640 10 monclient: monmap: 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:01.971 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.985+0000 7f0e750ab640 1 librados: starting msgr at 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.985+0000 7f0e750ab640 1 librados: starting objecter 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.985+0000 7f0e750ab640 1 Processor -- start 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.985+0000 7f0e750ab640 1 -- start start 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.985+0000 7f0e750ab640 1 librados: setting wanted keys 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.985+0000 7f0e750ab640 1 librados: calling monclient init 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.985+0000 7f0e750ab640 10 monclient: init 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 10 monclient(hunting): picked mon.noname-b con 0x7f0e68006a70 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:01.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 10 monclient(hunting): picked mon.noname-c con 0x7f0e68029c30 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:01.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 10 monclient(hunting): picked mon.noname-a con 0x7f0e68031b40 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:01.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e68164ec0 con 0x7f0e68031b40 2024-03-29T10:41:01.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e68164b90 con 0x7f0e68006a70 2024-03-29T10:41:01.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6802f2f0 con 0x7f0e68029c30 2024-03-29T10:41:01.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:01.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.986+0000 7f0e750ab640 10 monclient(hunting): authenticate will time out at 1183.248892s 2024-03-29T10:41:01.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.987+0000 7f0e6ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0e68006a70 0x7f0e68164480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:56684/0 (socket says 172.21.15.117:56684) 2024-03-29T10:41:01.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.987+0000 7f0e6ffff640 1 -- 172.21.15.117:0/226031511 learned_addr learned my addr 172.21.15.117:0/226031511 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 10 client.?.objecter ms_handle_connect 0x7f0e68006a70 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459350122 0 0) 0x7f0e68164b90 con 0x7f0e68006a70 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e54004eb0 con 0x7f0e68006a70 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 10 client.?.objecter ms_handle_connect 0x7f0e68029c30 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.990+0000 7f0e6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 client.?.objecter ms_handle_connect 0x7f0e68031b40 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401783913 0 0) 0x7f0e6802f2f0 con 0x7f0e68029c30 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:01.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e68164b90 con 0x7f0e68029c30 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2189578159 0 0) 0x7f0e54004eb0 con 0x7f0e68006a70 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e6802f2f0 con 0x7f0e68006a70 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359502320 0 0) 0x7f0e68164ec0 con 0x7f0e68031b40 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): my global_id is 4455 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e54004eb0 con 0x7f0e68031b40 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0e7004fdd0 con 0x7f0e68006a70 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:01.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): dump: 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1718427420 0 0) 0x7f0e68164b90 con 0x7f0e68029c30 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e68164ec0 con 0x7f0e68029c30 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0e5c001750 con 0x7f0e68029c30 2024-03-29T10:41:01.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 10 monclient(hunting): dump: 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1360605840 0 0) 0x7f0e54004eb0 con 0x7f0e68031b40 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e68164b90 con 0x7f0e68031b40 2024-03-29T10:41:01.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.991+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3284269482 0 0) 0x7f0e68164ec0 con 0x7f0e68029c30 2024-03-29T10:41:01.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 >> v1:172.21.15.136:6789/0 conn(0x7f0e68006a70 legacy=0x7f0e68164480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:01.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 >> v1:172.21.15.117:6789/0 conn(0x7f0e68031b40 legacy=0x7f0e6802d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:01.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:01.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 1 monclient: found mon.c 2024-03-29T10:41:01.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:01.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:01.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6816f800 con 0x7f0e68029c30 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 10 monclient: _finish_auth 0 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e750ab640 5 monclient: authenticate success, global_id 4505 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e750ab640 10 monclient: _renew_subs 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e750ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6816c7d0 con 0x7f0e68029c30 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e750ab640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e750ab640 10 monclient: _renew_subs 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e750ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6816cc60 con 0x7f0e68029c30 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.992+0000 7f0e750ab640 1 librados: init done 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e5c0018b0 con 0x7f0e68029c30 2024-03-29T10:41:01.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0e5c0035f0 con 0x7f0e68029c30 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e6cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e6cff9640 10 monclient: dump: 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:01.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e750ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:01.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e750ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:01.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e750ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:01.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e68004cd0 con 0x7f0e68029c30 2024-03-29T10:41:01.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.993+0000 7f0e6e7fc640 4 set_mon_vals no callback set 2024-03-29T10:41:01.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.997+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f0e5c003750 con 0x7f0e68029c30 2024-03-29T10:41:01.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.998+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f0e5c0835c0 con 0x7f0e68029c30 2024-03-29T10:41:01.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.998+0000 7f0e6cff9640 10 client.4505.objecter ms_dispatch 0x7f0e70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:01.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.998+0000 7f0e6cff9640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:01.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.998+0000 7f0e6cff9640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:01.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.998+0000 7f0e6cff9640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T10:41:01.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.998+0000 7f0e6ffff640 10 monclient: get_auth_request con 0x7f0e5406dfa0 auth_method 0 2024-03-29T10:41:01.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:01.998+0000 7f0e6cff9640 10 client.4505.objecter ms_handle_connect 0x7f0e5406dfa0 2024-03-29T10:41:01.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.001+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e5c04ecd0 con 0x7f0e68029c30 2024-03-29T10:41:01.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.001+0000 7f0e6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:01.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.001+0000 7f0e6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:02.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.325+0000 7f0e750ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:02.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.325+0000 7f0e750ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:02.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.325+0000 7f0e750ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:02.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.325+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e680c87a0 con 0x7f0e68029c30 2024-03-29T10:41:02.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.326+0000 7f0e6cff9640 1 -- 172.21.15.117:0/226031511 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f0e5c050dd0 con 0x7f0e68029c30 2024-03-29T10:41:02.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.326+0000 7f0e6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:02.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.326+0000 7f0e6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:02.310 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:41:02.310 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:02.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.329+0000 7f0e750ab640 10 librados: watch_flush enter 2024-03-29T10:41:02.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.329+0000 7f0e750ab640 10 librados: watch_flush exit 2024-03-29T10:41:02.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.329+0000 7f0e750ab640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T10:41:02.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.329+0000 7f0e750ab640 10 client.4505.objecter successfully canceled tick 2024-03-29T10:41:02.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.330+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 >> v1:172.21.15.117:6816/376768171 conn(0x7f0e5406dfa0 legacy=0x7f0e54070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:02.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.330+0000 7f0e750ab640 10 monclient: shutdown 2024-03-29T10:41:02.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.330+0000 7f0e750ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:02.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.330+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 >> v1:172.21.15.117:6790/0 conn(0x7f0e68029c30 legacy=0x7f0e6802cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:02.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.330+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 shutdown_connections 2024-03-29T10:41:02.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.330+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 >> 172.21.15.117:0/226031511 conn(0x7f0e680058f0 msgr2=0x7f0e680c3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:02.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.332+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 shutdown_connections 2024-03-29T10:41:02.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.332+0000 7f0e750ab640 1 -- 172.21.15.117:0/226031511 wait complete. 2024-03-29T10:41:02.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:02.332+0000 7f0e750ab640 1 librados: shutdown 2024-03-29T10:41:02.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:09.141 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:09.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.289+0000 7f63de163640 10 monclient: get_monmap_and_config 2024-03-29T10:41:09.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.289+0000 7f63de163640 10 monclient: build_initial_monmap 2024-03-29T10:41:09.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.289+0000 7f63de163640 10 monclient: monmap: 2024-03-29T10:41:09.273 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:09.273 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:09.273 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:41:09.290576+0000 2024-03-29T10:41:09.273 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:41:09.290576+0000 2024-03-29T10:41:09.273 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:09.274 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:09.274 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:09.274 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:41:09.274 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:41:09.274 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:09.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.290+0000 7f63de163640 1 Processor -- start 2024-03-29T10:41:09.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.290+0000 7f63de163640 1 -- start start 2024-03-29T10:41:09.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.290+0000 7f63de163640 10 monclient: init 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 10 monclient(hunting): picked mon.noname-b con 0x7f63d81457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 10 monclient(hunting): picked mon.noname-c con 0x7f63d813f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 10 monclient(hunting): picked mon.noname-a con 0x7f63d80a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80aaef0 con 0x7f63d80a9d70 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8145bd0 con 0x7f63d813f650 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8145eb0 con 0x7f63d81457f0 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63de163640 10 monclient(hunting): authenticate will time out at 1190.553764s 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63d77fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f63d81457f0 0x7f63d813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:39916/0 (socket says 172.21.15.117:39916) 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.291+0000 7f63d77fe640 1 -- 172.21.15.117:0/2471356266 learned_addr learned my addr 172.21.15.117:0/2471356266 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.292+0000 7f63d67fc640 1 -- 172.21.15.117:0/2471356266 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718634892 0 0) 0x7f63d8145eb0 con 0x7f63d81457f0 2024-03-29T10:41:09.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.292+0000 7f63d67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:09.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.292+0000 7f63d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:09.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.292+0000 7f63d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:09.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.292+0000 7f63d67fc640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:41:09.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.292+0000 7f63d67fc640 1 -- 172.21.15.117:0/2471356266 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c0003520 con 0x7f63d81457f0 2024-03-29T10:41:09.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.292+0000 7f63d67fc640 1 -- 172.21.15.117:0/2471356266 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 501434724 0 0) 0x7f63c0003520 con 0x7f63d81457f0 2024-03-29T10:41:09.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.292+0000 7f63d67fc640 1 -- 172.21.15.117:0/2471356266 >> v1:172.21.15.136:6789/0 conn(0x7f63d813f650 legacy=0x7f63d80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:09.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.292+0000 7f63d67fc640 1 -- 172.21.15.117:0/2471356266 >> v1:172.21.15.117:6789/0 conn(0x7f63d80a9d70 legacy=0x7f63d814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:09.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:09.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 1 monclient: found mon.noname-b 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 1 -- 172.21.15.117:0/2471356266 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d8146340 con 0x7f63d81457f0 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 1 -- 172.21.15.117:0/2471356266 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f63c8001690 con 0x7f63d81457f0 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:09.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63d67fc640 10 monclient: dump: 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63de163640 5 monclient: authenticate success, global_id 4508 2024-03-29T10:41:09.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.295+0000 7f63de163640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:09.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.296+0000 7f63d67fc640 1 -- 172.21.15.117:0/2471356266 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63c80018f0 con 0x7f63d81457f0 2024-03-29T10:41:09.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.296+0000 7f63d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.296+0000 7f63d67fc640 1 -- 172.21.15.117:0/2471356266 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f63c8001d40 con 0x7f63d81457f0 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.296+0000 7f63de163640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.296+0000 7f63de163640 4 set_mon_vals no callback set 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.296+0000 7f63d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.296+0000 7f63d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.296+0000 7f63d67fc640 10 monclient: dump: 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:09.282 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.296+0000 7f63de163640 10 monclient: shutdown 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.297+0000 7f63de163640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.297+0000 7f63de163640 1 -- 172.21.15.117:0/2471356266 >> v1:172.21.15.117:6790/0 conn(0x7f63d81457f0 legacy=0x7f63d813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.297+0000 7f63de163640 1 -- 172.21.15.117:0/2471356266 shutdown_connections 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.297+0000 7f63de163640 1 -- 172.21.15.117:0/2471356266 >> 172.21.15.117:0/2471356266 conn(0x7f63d8066a00 msgr2=0x7f63d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.297+0000 7f63de163640 1 -- 172.21.15.117:0/2471356266 shutdown_connections 2024-03-29T10:41:09.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.297+0000 7f63de163640 1 -- 172.21.15.117:0/2471356266 wait complete. 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.298+0000 7f63de163640 10 monclient: build_initial_monmap 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.298+0000 7f63de163640 10 monclient: monmap: 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:09.290 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.299+0000 7f63de163640 1 librados: starting msgr at 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.299+0000 7f63de163640 1 librados: starting objecter 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.299+0000 7f63de163640 1 Processor -- start 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.299+0000 7f63de163640 1 -- start start 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.299+0000 7f63de163640 1 librados: setting wanted keys 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.299+0000 7f63de163640 1 librados: calling monclient init 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.299+0000 7f63de163640 10 monclient: init 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 10 monclient(hunting): picked mon.noname-b con 0x7f63d80a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:09.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 10 monclient(hunting): picked mon.noname-c con 0x7f63d813f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 10 monclient(hunting): picked mon.noname-a con 0x7f63d81457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80716f0 con 0x7f63d81457f0 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8136780 con 0x7f63d80a9d70 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8071960 con 0x7f63d813f650 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.300+0000 7f63de163640 10 monclient(hunting): authenticate will time out at 1190.563082s 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.301+0000 7f63d77fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f63d80a9d70 0x7f63d8070410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:34088/0 (socket says 172.21.15.117:34088) 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.301+0000 7f63d77fe640 1 -- 172.21.15.117:0/2878502083 learned_addr learned my addr 172.21.15.117:0/2878502083 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.301+0000 7f63bffff640 10 client.?.objecter ms_handle_connect 0x7f63d80a9d70 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.301+0000 7f63bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.301+0000 7f63bffff640 10 client.?.objecter ms_handle_connect 0x7f63d813f650 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.301+0000 7f63bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981013355 0 0) 0x7f63d8136780 con 0x7f63d80a9d70 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c4004eb0 con 0x7f63d80a9d70 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880342126 0 0) 0x7f63d8071960 con 0x7f63d813f650 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d8136780 con 0x7f63d813f650 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.302+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3913876426 0 0) 0x7f63d8136780 con 0x7f63d813f650 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63d8071960 con 0x7f63d813f650 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f63cc001760 con 0x7f63d813f650 2024-03-29T10:41:09.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient(hunting): dump: 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:09.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2187398176 0 0) 0x7f63c4004eb0 con 0x7f63d80a9d70 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63d8136780 con 0x7f63d80a9d70 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f63c8001650 con 0x7f63d80a9d70 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient(hunting): dump: 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:09.295 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3578184054 0 0) 0x7f63d8071960 con 0x7f63d813f650 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 >> v1:172.21.15.136:6789/0 conn(0x7f63d80a9d70 legacy=0x7f63d8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 >> v1:172.21.15.117:6789/0 conn(0x7f63d81457f0 legacy=0x7f63d8136070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 monclient: found mon.c 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d8071c40 con 0x7f63d813f650 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient: _finish_auth 0 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.303+0000 7f63bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63de163640 5 monclient: authenticate success, global_id 4511 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63de163640 10 monclient: _renew_subs 2024-03-29T10:41:09.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63de163640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d8070b20 con 0x7f63d813f650 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63de163640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63de163640 10 monclient: _renew_subs 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63de163640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d81e7340 con 0x7f63d813f650 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63de163640 1 librados: init done 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63cc001c70 con 0x7f63d813f650 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63d5ffb640 4 set_mon_vals no callback set 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f63cc003790 con 0x7f63d813f650 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.304+0000 7f63bffff640 10 monclient: dump: 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.305+0000 7f63de163640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.305+0000 7f63de163640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.306+0000 7f63de163640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.306+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b8005080 con 0x7f63d813f650 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.306+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f63cc019760 con 0x7f63d813f650 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.307+0000 7f63d77fe640 10 monclient: get_auth_request con 0x7f63c406dfb0 auth_method 0 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.307+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f63cc0533a0 con 0x7f63d813f650 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.307+0000 7f63bffff640 10 client.4511.objecter ms_dispatch 0x7f63d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.307+0000 7f63bffff640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.307+0000 7f63bffff640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.307+0000 7f63bffff640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.307+0000 7f63bffff640 10 client.4511.objecter ms_handle_connect 0x7f63c406dfb0 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.316+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63cc04f640 con 0x7f63d813f650 2024-03-29T10:41:09.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.316+0000 7f63bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:09.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.316+0000 7f63bffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:09.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.619+0000 7f63de163640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:09.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.619+0000 7f63de163640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:09.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.619+0000 7f63de163640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:09.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.619+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63b8005330 con 0x7f63d813f650 2024-03-29T10:41:09.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.620+0000 7f63bffff640 1 -- 172.21.15.117:0/2878502083 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f63cc04f7e0 con 0x7f63d813f650 2024-03-29T10:41:09.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.620+0000 7f63bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:09.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.620+0000 7f63bffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:09.604 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:41:09.604 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:09.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.623+0000 7f63de163640 10 librados: watch_flush enter 2024-03-29T10:41:09.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.623+0000 7f63de163640 10 librados: watch_flush exit 2024-03-29T10:41:09.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.623+0000 7f63de163640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T10:41:09.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.623+0000 7f63de163640 10 client.4511.objecter successfully canceled tick 2024-03-29T10:41:09.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.624+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 >> v1:172.21.15.117:6816/376768171 conn(0x7f63c406dfb0 legacy=0x7f63c4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:09.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.624+0000 7f63de163640 10 monclient: shutdown 2024-03-29T10:41:09.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.624+0000 7f63de163640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:09.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.624+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 >> v1:172.21.15.117:6790/0 conn(0x7f63d813f650 legacy=0x7f63d8135960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:09.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.624+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 shutdown_connections 2024-03-29T10:41:09.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.624+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 >> 172.21.15.117:0/2878502083 conn(0x7f63d8066a00 msgr2=0x7f63d80b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:09.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.626+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 shutdown_connections 2024-03-29T10:41:09.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.626+0000 7f63de163640 1 -- 172.21.15.117:0/2878502083 wait complete. 2024-03-29T10:41:09.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:09.626+0000 7f63de163640 1 librados: shutdown 2024-03-29T10:41:09.623 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:16.430 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:16.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.598+0000 7f1d6aac9640 10 monclient: get_monmap_and_config 2024-03-29T10:41:16.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.598+0000 7f1d6aac9640 10 monclient: build_initial_monmap 2024-03-29T10:41:16.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.598+0000 7f1d6aac9640 10 monclient: monmap: 2024-03-29T10:41:16.582 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:16.583 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:16.583 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:41:16.599763+0000 2024-03-29T10:41:16.583 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:41:16.599763+0000 2024-03-29T10:41:16.583 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:16.583 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:16.583 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:16.583 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:41:16.583 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:41:16.583 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.599+0000 7f1d6aac9640 1 Processor -- start 2024-03-29T10:41:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.600+0000 7f1d6aac9640 1 -- start start 2024-03-29T10:41:16.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.600+0000 7f1d6aac9640 10 monclient: init 2024-03-29T10:41:16.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.601+0000 7f1d6aac9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.601+0000 7f1d6aac9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.601+0000 7f1d68865640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f1d640b0860 0x7f1d6413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:50932/0 (socket says 172.21.15.117:50932) 2024-03-29T10:41:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.601+0000 7f1d68865640 1 -- 172.21.15.117:0/3502918540 learned_addr learned my addr 172.21.15.117:0/3502918540 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.601+0000 7f1d6aac9640 10 monclient(hunting): picked mon.noname-b con 0x7f1d640b0860 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:16.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.601+0000 7f1d6aac9640 10 monclient(hunting): picked mon.noname-a con 0x7f1d6413ef30 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d6aac9640 10 monclient(hunting): picked mon.noname-c con 0x7f1d640af8e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d6aac9640 1 -- 172.21.15.117:0/3502918540 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d640b0c40 con 0x7f1d6413ef30 2024-03-29T10:41:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d6aac9640 1 -- 172.21.15.117:0/3502918540 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d640aed10 con 0x7f1d640af8e0 2024-03-29T10:41:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d6aac9640 1 -- 172.21.15.117:0/3502918540 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d640afd50 con 0x7f1d640b0860 2024-03-29T10:41:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d6aac9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d6aac9640 10 monclient(hunting): authenticate will time out at 1197.869463s 2024-03-29T10:41:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d637fe640 1 -- 172.21.15.117:0/3502918540 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213104576 0 0) 0x7f1d640b0c40 con 0x7f1d6413ef30 2024-03-29T10:41:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:16.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d637fe640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.607+0000 7f1d637fe640 1 -- 172.21.15.117:0/3502918540 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d54007930 con 0x7f1d6413ef30 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 1 -- 172.21.15.117:0/3502918540 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3749380381 0 0) 0x7f1d54007930 con 0x7f1d6413ef30 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 1 -- 172.21.15.117:0/3502918540 >> v1:172.21.15.117:6790/0 conn(0x7f1d640b0860 legacy=0x7f1d6413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 1 -- 172.21.15.117:0/3502918540 >> v1:172.21.15.136:6789/0 conn(0x7f1d640af8e0 legacy=0x7f1d64149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 1 monclient: found mon.noname-a 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 1 -- 172.21.15.117:0/3502918540 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d640ab1b0 con 0x7f1d6413ef30 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:16.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 1 -- 172.21.15.117:0/3502918540 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1d50001690 con 0x7f1d6413ef30 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: dump: 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:16.598 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d6aac9640 5 monclient: authenticate success, global_id 4458 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d6aac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 1 -- 172.21.15.117:0/3502918540 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d50001860 con 0x7f1d6413ef30 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 1 -- 172.21.15.117:0/3502918540 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1d50001c70 con 0x7f1d6413ef30 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d637fe640 10 monclient: dump: 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:16.599 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d6aac9640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.608+0000 7f1d6aac9640 4 set_mon_vals no callback set 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.609+0000 7f1d6aac9640 10 monclient: shutdown 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.609+0000 7f1d6aac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.609+0000 7f1d6aac9640 1 -- 172.21.15.117:0/3502918540 >> v1:172.21.15.117:6789/0 conn(0x7f1d6413ef30 legacy=0x7f1d640af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.609+0000 7f1d6aac9640 1 -- 172.21.15.117:0/3502918540 shutdown_connections 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.610+0000 7f1d6aac9640 1 -- 172.21.15.117:0/3502918540 >> 172.21.15.117:0/3502918540 conn(0x7f1d6406b660 msgr2=0x7f1d6406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.610+0000 7f1d6aac9640 1 -- 172.21.15.117:0/3502918540 shutdown_connections 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.610+0000 7f1d6aac9640 1 -- 172.21.15.117:0/3502918540 wait complete. 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.611+0000 7f1d6aac9640 10 monclient: build_initial_monmap 2024-03-29T10:41:16.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.611+0000 7f1d6aac9640 10 monclient: monmap: 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:41:16.601 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.612+0000 7f1d6aac9640 1 librados: starting msgr at 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.612+0000 7f1d6aac9640 1 librados: starting objecter 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.612+0000 7f1d6aac9640 1 Processor -- start 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.613+0000 7f1d6aac9640 1 -- start start 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.614+0000 7f1d6aac9640 1 librados: setting wanted keys 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.614+0000 7f1d6aac9640 1 librados: calling monclient init 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.614+0000 7f1d6aac9640 10 monclient: init 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:16.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 10 monclient(hunting): picked mon.noname-b con 0x7f1d640af8e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 10 monclient(hunting): picked mon.noname-a con 0x7f1d640b0860 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 10 monclient(hunting): picked mon.noname-c con 0x7f1d6413ef30 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6406deb0 con 0x7f1d640b0860 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6406ec50 con 0x7f1d640af8e0 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d640b3770 con 0x7f1d6413ef30 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d6aac9640 10 monclient(hunting): authenticate will time out at 1197.877557s 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d69066640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f1d6413ef30 0x7f1d641dd790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41358/0 (socket says 172.21.15.117:41358) 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.615+0000 7f1d69066640 1 -- 172.21.15.117:0/1038527755 learned_addr learned my addr 172.21.15.117:0/1038527755 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.616+0000 7f1d617fa640 10 client.?.objecter ms_handle_connect 0x7f1d6413ef30 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.616+0000 7f1d617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.616+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638769876 0 0) 0x7f1d640b3770 con 0x7f1d6413ef30 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.616+0000 7f1d617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.616+0000 7f1d617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.616+0000 7f1d617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.616+0000 7f1d617fa640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.616+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d48004eb0 con 0x7f1d6413ef30 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.617+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4125013983 0 0) 0x7f1d48004eb0 con 0x7f1d6413ef30 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.618+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d640b3770 con 0x7f1d6413ef30 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.618+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1d5c001b70 con 0x7f1d6413ef30 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.618+0000 7f1d617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.618+0000 7f1d617fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.618+0000 7f1d617fa640 10 monclient(hunting): dump: 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.618+0000 7f1d617fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.618+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3337865329 0 0) 0x7f1d640b3770 con 0x7f1d6413ef30 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.618+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 >> v1:172.21.15.136:6789/0 conn(0x7f1d640af8e0 legacy=0x7f1d640b3db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 >> v1:172.21.15.117:6789/0 conn(0x7f1d640b0860 legacy=0x7f1d641d2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 10 monclient: _finish_hunting 0 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 1 monclient: found mon.c 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d640b3900 con 0x7f1d6413ef30 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 10 monclient: _finish_auth 0 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 10 monclient: _check_auth_tickets 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 10 client.?.objecter ms_handle_connect 0x7f1d640b0860 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d6aac9640 5 monclient: authenticate success, global_id 4514 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d6aac9640 10 monclient: _renew_subs 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d6aac9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.620+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6406d9d0 con 0x7f1d6413ef30 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d6aac9640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d5c001cd0 con 0x7f1d6413ef30 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1d5c004e20 con 0x7f1d6413ef30 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d617fa640 10 monclient: dump: 2024-03-29T10:41:16.613 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d62ffd640 4 set_mon_vals no callback set 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d6aac9640 10 monclient: _renew_subs 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d6aac9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d641d35b0 con 0x7f1d6413ef30 2024-03-29T10:41:16.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.621+0000 7f1d6aac9640 1 librados: init done 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.623+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f1d5c004f80 con 0x7f1d6413ef30 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.623+0000 7f1d6aac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.623+0000 7f1d6aac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.624+0000 7f1d6aac9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.624+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d640ab770 con 0x7f1d6413ef30 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.624+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f1d5c084dc0 con 0x7f1d6413ef30 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.624+0000 7f1d617fa640 10 client.4514.objecter ms_dispatch 0x7f1d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.624+0000 7f1d617fa640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.624+0000 7f1d617fa640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.624+0000 7f1d617fa640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.624+0000 7f1d68865640 10 monclient: get_auth_request con 0x7f1d4806db80 auth_method 0 2024-03-29T10:41:16.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.625+0000 7f1d617fa640 10 client.4514.objecter ms_handle_connect 0x7f1d4806db80 2024-03-29T10:41:16.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.632+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d640ab770 con 0x7f1d6413ef30 2024-03-29T10:41:16.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.632+0000 7f1d617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:16.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.632+0000 7f1d617fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:16.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.956+0000 7f1d6aac9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:16.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.956+0000 7f1d6aac9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:16.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.956+0000 7f1d6aac9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:16.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.956+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d640ab770 con 0x7f1d6413ef30 2024-03-29T10:41:16.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.959+0000 7f1d617fa640 1 -- 172.21.15.117:0/1038527755 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f1d640ab770 con 0x7f1d6413ef30 2024-03-29T10:41:16.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.959+0000 7f1d617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:16.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.959+0000 7f1d617fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:16.945 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:41:16.945 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:16.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.964+0000 7f1d6aac9640 10 librados: watch_flush enter 2024-03-29T10:41:16.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.966+0000 7f1d6aac9640 10 librados: watch_flush exit 2024-03-29T10:41:16.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.966+0000 7f1d6aac9640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T10:41:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.966+0000 7f1d6aac9640 10 client.4514.objecter successfully canceled tick 2024-03-29T10:41:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.966+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 >> v1:172.21.15.117:6816/376768171 conn(0x7f1d4806db80 legacy=0x7f1d48070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.966+0000 7f1d6aac9640 10 monclient: shutdown 2024-03-29T10:41:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.966+0000 7f1d6aac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.966+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 >> v1:172.21.15.117:6790/0 conn(0x7f1d6413ef30 legacy=0x7f1d641dd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.966+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 shutdown_connections 2024-03-29T10:41:16.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.966+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 >> 172.21.15.117:0/1038527755 conn(0x7f1d6406b660 msgr2=0x7f1d6406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:16.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.968+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 shutdown_connections 2024-03-29T10:41:16.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.968+0000 7f1d6aac9640 1 -- 172.21.15.117:0/1038527755 wait complete. 2024-03-29T10:41:16.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:16.968+0000 7f1d6aac9640 1 librados: shutdown 2024-03-29T10:41:16.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:23.776 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:23.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.940+0000 7f62a48f6640 10 monclient: get_monmap_and_config 2024-03-29T10:41:23.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.940+0000 7f62a48f6640 10 monclient: build_initial_monmap 2024-03-29T10:41:23.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.940+0000 7f62a48f6640 10 monclient: monmap: 2024-03-29T10:41:23.924 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:23.924 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:23.924 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:41:23.941759+0000 2024-03-29T10:41:23.925 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:41:23.941759+0000 2024-03-29T10:41:23.925 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:23.925 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:23.925 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:23.925 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:41:23.925 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:41:23.925 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:23.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.942+0000 7f62a48f6640 1 Processor -- start 2024-03-29T10:41:23.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.942+0000 7f62a48f6640 1 -- start start 2024-03-29T10:41:23.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.942+0000 7f62a48f6640 10 monclient: init 2024-03-29T10:41:23.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:23.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:23.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 10 monclient(hunting): picked mon.noname-c con 0x7f62a01457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:23.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 10 monclient(hunting): picked mon.noname-a con 0x7f62a013f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:23.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 10 monclient(hunting): picked mon.noname-b con 0x7f62a00a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:23.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a00aaef0 con 0x7f62a013f650 2024-03-29T10:41:23.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a0145bd0 con 0x7f62a01457f0 2024-03-29T10:41:23.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a0145eb0 con 0x7f62a00a9d70 2024-03-29T10:41:23.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:23.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.943+0000 7f62a48f6640 10 monclient(hunting): authenticate will time out at 1205.205265s 2024-03-29T10:41:23.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629ed9d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f62a00a9d70 0x7f62a014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:53158/0 (socket says 172.21.15.117:53158) 2024-03-29T10:41:23.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629ed9d640 1 -- 172.21.15.117:0/4207052157 learned_addr learned my addr 172.21.15.117:0/4207052157 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:23.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256711795 0 0) 0x7f62a0145bd0 con 0x7f62a01457f0 2024-03-29T10:41:23.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:23.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:23.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:23.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:41:23.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6288003520 con 0x7f62a01457f0 2024-03-29T10:41:23.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535377923 0 0) 0x7f62a00aaef0 con 0x7f62a013f650 2024-03-29T10:41:23.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:23.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.946+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a0145bd0 con 0x7f62a013f650 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2162838219 0 0) 0x7f62a0145bd0 con 0x7f62a013f650 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 >> v1:172.21.15.117:6790/0 conn(0x7f62a00a9d70 legacy=0x7f62a014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 >> v1:172.21.15.136:6789/0 conn(0x7f62a01457f0 legacy=0x7f62a013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 1 monclient: found mon.noname-a 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a0146340 con 0x7f62a013f650 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: _finish_auth 0 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:41:23.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6294001670 con 0x7f62a013f650 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: dump: 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:23.934 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f62a48f6640 5 monclient: authenticate success, global_id 4464 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f62a48f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62940017d0 con 0x7f62a013f650 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f62a48f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f62a48f6640 4 set_mon_vals no callback set 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 1 -- 172.21.15.117:0/4207052157 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6294001c20 con 0x7f62a013f650 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:23.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.947+0000 7f629d59a640 10 monclient: dump: 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.948+0000 7f62a48f6640 10 monclient: shutdown 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.948+0000 7f62a48f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:23.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.948+0000 7f62a48f6640 1 -- 172.21.15.117:0/4207052157 >> v1:172.21.15.117:6789/0 conn(0x7f62a013f650 legacy=0x7f62a00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.950+0000 7f62a48f6640 1 -- 172.21.15.117:0/4207052157 shutdown_connections 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.950+0000 7f62a48f6640 1 -- 172.21.15.117:0/4207052157 >> 172.21.15.117:0/4207052157 conn(0x7f62a0066a00 msgr2=0x7f62a0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.951+0000 7f62a48f6640 1 -- 172.21.15.117:0/4207052157 shutdown_connections 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.951+0000 7f62a48f6640 1 -- 172.21.15.117:0/4207052157 wait complete. 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.952+0000 7f62a48f6640 10 monclient: build_initial_monmap 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.952+0000 7f62a48f6640 10 monclient: monmap: 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:41:23.939 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.953+0000 7f62a48f6640 1 librados: starting msgr at 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.953+0000 7f62a48f6640 1 librados: starting objecter 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.953+0000 7f62a48f6640 1 Processor -- start 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.953+0000 7f62a48f6640 1 -- start start 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.953+0000 7f62a48f6640 1 librados: setting wanted keys 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.953+0000 7f62a48f6640 1 librados: calling monclient init 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.953+0000 7f62a48f6640 10 monclient: init 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:23.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 10 monclient(hunting): picked mon.noname-c con 0x7f62a00a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 10 monclient(hunting): picked mon.noname-b con 0x7f62a013f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 10 monclient(hunting): picked mon.noname-a con 0x7f62a01457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a006f240 con 0x7f62a01457f0 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a0070ee0 con 0x7f62a013f650 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a00711c0 con 0x7f62a00a9d70 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f62a48f6640 10 monclient(hunting): authenticate will time out at 1205.216677s 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f629e59c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f62a00a9d70 0x7f62a006f6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:59490/0 (socket says 172.21.15.117:59490) 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.954+0000 7f629e59c640 1 -- 172.21.15.117:0/1200171123 learned_addr learned my addr 172.21.15.117:0/1200171123 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.955+0000 7f628effd640 10 client.?.objecter ms_handle_connect 0x7f62a00a9d70 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.955+0000 7f628effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.955+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574951700 0 0) 0x7f62a00711c0 con 0x7f62a00a9d70 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.955+0000 7f628effd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.956+0000 7f628effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:23.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.956+0000 7f628effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:23.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.956+0000 7f628effd640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:41:23.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.956+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6280004eb0 con 0x7f62a00a9d70 2024-03-29T10:41:23.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.958+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3010747010 0 0) 0x7f6280004eb0 con 0x7f62a00a9d70 2024-03-29T10:41:23.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.958+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62a00711c0 con 0x7f62a00a9d70 2024-03-29T10:41:23.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.958+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6290001990 con 0x7f62a00a9d70 2024-03-29T10:41:23.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.958+0000 7f628effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:23.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.958+0000 7f628effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:23.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.958+0000 7f628effd640 10 monclient(hunting): dump: 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.958+0000 7f628effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.958+0000 7f628effd640 10 client.?.objecter ms_handle_connect 0x7f62a013f650 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.958+0000 7f628effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 48845357 0 0) 0x7f62a00711c0 con 0x7f62a00a9d70 2024-03-29T10:41:23.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 >> v1:172.21.15.136:6789/0 conn(0x7f62a013f650 legacy=0x7f62a006fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 >> v1:172.21.15.117:6789/0 conn(0x7f62a01457f0 legacy=0x7f62a01e5fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 1 monclient: found mon.c 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a013a1c0 con 0x7f62a00a9d70 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 10 monclient: _finish_auth 0 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f628effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f62a48f6640 5 monclient: authenticate success, global_id 4520 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f62a48f6640 10 monclient: _renew_subs 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f62a48f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.959+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a006edd0 con 0x7f62a00a9d70 2024-03-29T10:41:23.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f62a48f6640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f62a48f6640 10 monclient: _renew_subs 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f62a48f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a0071e10 con 0x7f62a00a9d70 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f62a48f6640 1 librados: init done 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6290001c90 con 0x7f62a00a9d70 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f628effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6290003a40 con 0x7f62a00a9d70 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f628effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f628effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:23.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f628effd640 10 monclient: dump: 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:23.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.960+0000 7f629cd99640 4 set_mon_vals no callback set 2024-03-29T10:41:23.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.964+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f6290003ba0 con 0x7f62a00a9d70 2024-03-29T10:41:23.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.964+0000 7f62a48f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:23.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.965+0000 7f629dd9b640 10 monclient: get_auth_request con 0x7f628006dcb0 auth_method 0 2024-03-29T10:41:23.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.965+0000 7f628effd640 10 client.4520.objecter ms_handle_connect 0x7f628006dcb0 2024-03-29T10:41:23.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.966+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f6290084b20 con 0x7f62a00a9d70 2024-03-29T10:41:23.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.966+0000 7f628effd640 10 client.4520.objecter ms_dispatch 0x7f62a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:23.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.966+0000 7f628effd640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:23.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.966+0000 7f628effd640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:23.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.966+0000 7f628effd640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T10:41:23.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.966+0000 7f62a48f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:23.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.966+0000 7f62a48f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:23.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.966+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6284005080 con 0x7f62a00a9d70 2024-03-29T10:41:23.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.975+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6290050400 con 0x7f62a00a9d70 2024-03-29T10:41:23.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.975+0000 7f628effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:23.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:23.975+0000 7f628effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:24.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.312+0000 7f62a48f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:24.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.312+0000 7f62a48f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:24.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.312+0000 7f62a48f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:24.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.312+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6284005330 con 0x7f62a00a9d70 2024-03-29T10:41:24.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.313+0000 7f628effd640 1 -- 172.21.15.117:0/1200171123 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f6290052330 con 0x7f62a00a9d70 2024-03-29T10:41:24.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.313+0000 7f628effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:24.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.313+0000 7f628effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:24.297 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:41:24.297 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:24.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.316+0000 7f62a48f6640 10 librados: watch_flush enter 2024-03-29T10:41:24.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.316+0000 7f62a48f6640 10 librados: watch_flush exit 2024-03-29T10:41:24.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.316+0000 7f62a48f6640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T10:41:24.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.316+0000 7f62a48f6640 10 client.4520.objecter successfully canceled tick 2024-03-29T10:41:24.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.317+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 >> v1:172.21.15.117:6816/376768171 conn(0x7f628006dcb0 legacy=0x7f6280070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:24.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.317+0000 7f62a48f6640 10 monclient: shutdown 2024-03-29T10:41:24.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.317+0000 7f62a48f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:24.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.317+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 >> v1:172.21.15.117:6790/0 conn(0x7f62a00a9d70 legacy=0x7f62a006f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:24.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.317+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 shutdown_connections 2024-03-29T10:41:24.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.317+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 >> 172.21.15.117:0/1200171123 conn(0x7f62a0066a00 msgr2=0x7f62a013abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:24.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.317+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 shutdown_connections 2024-03-29T10:41:24.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.317+0000 7f62a48f6640 1 -- 172.21.15.117:0/1200171123 wait complete. 2024-03-29T10:41:24.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:24.321+0000 7f62a48f6640 1 librados: shutdown 2024-03-29T10:41:24.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:31.129 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:31.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.294+0000 7fdf7ab3b640 10 monclient: get_monmap_and_config 2024-03-29T10:41:31.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.294+0000 7fdf7ab3b640 10 monclient: build_initial_monmap 2024-03-29T10:41:31.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.294+0000 7fdf7ab3b640 10 monclient: monmap: 2024-03-29T10:41:31.278 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:31.278 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:31.279 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:41:31.295840+0000 2024-03-29T10:41:31.279 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:41:31.295840+0000 2024-03-29T10:41:31.279 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:31.279 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:31.279 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:31.279 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:41:31.279 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:41:31.279 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:31.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.296+0000 7fdf7ab3b640 1 Processor -- start 2024-03-29T10:41:31.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.298+0000 7fdf7ab3b640 1 -- start start 2024-03-29T10:41:31.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.298+0000 7fdf7ab3b640 10 monclient: init 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 10 monclient(hunting): picked mon.noname-c con 0x7fdf6c038c60 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 10 monclient(hunting): picked mon.noname-a con 0x7fdf6c0324b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 10 monclient(hunting): picked mon.noname-b con 0x7fdf6c0d11e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c0caf80 con 0x7fdf6c0324b0 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c039850 con 0x7fdf6c038c60 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c0c78d0 con 0x7fdf6c0d11e0 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:31.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7ab3b640 10 monclient(hunting): authenticate will time out at 1212.561207s 2024-03-29T10:41:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.299+0000 7fdf7a33a640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fdf6c0d11e0 0x7fdf6c0d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:39264/0 (socket says 172.21.15.117:39264) 2024-03-29T10:41:31.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.300+0000 7fdf7a33a640 1 -- 172.21.15.117:0/3018428589 learned_addr learned my addr 172.21.15.117:0/3018428589 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.302+0000 7fdf78b37640 1 -- 172.21.15.117:0/3018428589 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348203850 0 0) 0x7fdf6c0c78d0 con 0x7fdf6c0d11e0 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.302+0000 7fdf78b37640 10 monclient(hunting): _negotiate 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.302+0000 7fdf78b37640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.302+0000 7fdf78b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.302+0000 7fdf78b37640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.302+0000 7fdf78b37640 1 -- 172.21.15.117:0/3018428589 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf60003520 con 0x7fdf6c0d11e0 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.303+0000 7fdf78b37640 1 -- 172.21.15.117:0/3018428589 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1550861002 0 0) 0x7fdf60003520 con 0x7fdf6c0d11e0 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.303+0000 7fdf78b37640 1 -- 172.21.15.117:0/3018428589 >> v1:172.21.15.136:6789/0 conn(0x7fdf6c038c60 legacy=0x7fdf6c0c4ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.303+0000 7fdf78b37640 1 -- 172.21.15.117:0/3018428589 >> v1:172.21.15.117:6789/0 conn(0x7fdf6c0324b0 legacy=0x7fdf6c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.303+0000 7fdf78b37640 10 monclient: _finish_hunting 0 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.303+0000 7fdf78b37640 1 monclient: found mon.noname-b 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.303+0000 7fdf78b37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.303+0000 7fdf78b37640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:41:31.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.303+0000 7fdf78b37640 1 -- 172.21.15.117:0/3018428589 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c0c7a60 con 0x7fdf6c0d11e0 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.303+0000 7fdf78b37640 10 monclient: _finish_auth 0 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.304+0000 7fdf78b37640 10 monclient: _check_auth_tickets 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.304+0000 7fdf78b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.304+0000 7fdf78b37640 1 -- 172.21.15.117:0/3018428589 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdf64001690 con 0x7fdf6c0d11e0 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.304+0000 7fdf78b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.304+0000 7fdf78b37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.304+0000 7fdf78b37640 10 monclient: dump: 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:31.290 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.304+0000 7fdf7ab3b640 5 monclient: authenticate success, global_id 4523 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.304+0000 7fdf7ab3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.305+0000 7fdf78b37640 1 -- 172.21.15.117:0/3018428589 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf640019a0 con 0x7fdf6c0d11e0 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.305+0000 7fdf78b37640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.305+0000 7fdf78b37640 1 -- 172.21.15.117:0/3018428589 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdf64001d30 con 0x7fdf6c0d11e0 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.305+0000 7fdf78b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.305+0000 7fdf78b37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.305+0000 7fdf78b37640 10 monclient: dump: 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:31.291 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:31.292 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:31.292 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:31.292 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:31.292 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:31.292 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:31.292 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:31.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.305+0000 7fdf7ab3b640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:31.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.305+0000 7fdf7ab3b640 4 set_mon_vals no callback set 2024-03-29T10:41:31.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.306+0000 7fdf7ab3b640 10 monclient: shutdown 2024-03-29T10:41:31.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.306+0000 7fdf7ab3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:31.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.306+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/3018428589 >> v1:172.21.15.117:6790/0 conn(0x7fdf6c0d11e0 legacy=0x7fdf6c0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:31.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.307+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/3018428589 shutdown_connections 2024-03-29T10:41:31.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.307+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/3018428589 >> 172.21.15.117:0/3018428589 conn(0x7fdf6c005c20 msgr2=0x7fdf6c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:31.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.310+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/3018428589 shutdown_connections 2024-03-29T10:41:31.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.310+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/3018428589 wait complete. 2024-03-29T10:41:31.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.311+0000 7fdf7ab3b640 10 monclient: build_initial_monmap 2024-03-29T10:41:31.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.312+0000 7fdf7ab3b640 10 monclient: monmap: 2024-03-29T10:41:31.295 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:31.295 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:31.295 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:41:31.295 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:41:31.295 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:31.295 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:31.296 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:31.296 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:41:31.296 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:41:31.296 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:31.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.312+0000 7fdf7ab3b640 1 librados: starting msgr at 2024-03-29T10:41:31.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.312+0000 7fdf7ab3b640 1 librados: starting objecter 2024-03-29T10:41:31.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.312+0000 7fdf7ab3b640 1 Processor -- start 2024-03-29T10:41:31.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.313+0000 7fdf7ab3b640 1 -- start start 2024-03-29T10:41:31.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.313+0000 7fdf7ab3b640 1 librados: setting wanted keys 2024-03-29T10:41:31.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.313+0000 7fdf7ab3b640 1 librados: calling monclient init 2024-03-29T10:41:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.313+0000 7fdf7ab3b640 10 monclient: init 2024-03-29T10:41:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.314+0000 7fdf7ab3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.314+0000 7fdf7ab3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.314+0000 7fdf7ab3b640 10 monclient(hunting): picked mon.noname-c con 0x7fdf6c0324b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:31.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.314+0000 7fdf7ab3b640 10 monclient(hunting): picked mon.noname-a con 0x7fdf6c038c60 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf7ab3b640 10 monclient(hunting): picked mon.noname-b con 0x7fdf6c0d11e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf7ab3b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c036860 con 0x7fdf6c038c60 2024-03-29T10:41:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf7ab3b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c0c2460 con 0x7fdf6c0d11e0 2024-03-29T10:41:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf7ab3b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c0c2740 con 0x7fdf6c0324b0 2024-03-29T10:41:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf7ab3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf7ab3b640 10 monclient(hunting): authenticate will time out at 1212.578441s 2024-03-29T10:41:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf79b39640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fdf6c0324b0 0x7fdf6c0355e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:43454/0 (socket says 172.21.15.117:43454) 2024-03-29T10:41:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf79b39640 1 -- 172.21.15.117:0/4252903666 learned_addr learned my addr 172.21.15.117:0/4252903666 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:31.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf6a7fc640 10 client.?.objecter ms_handle_connect 0x7fdf6c0324b0 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900534446 0 0) 0x7fdf6c0c2740 con 0x7fdf6c0324b0 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf6a7fc640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.316+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf58004eb0 con 0x7fdf6c0324b0 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1457329690 0 0) 0x7fdf58004eb0 con 0x7fdf6c0324b0 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf58005300 con 0x7fdf6c0324b0 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdf70001830 con 0x7fdf6c0324b0 2024-03-29T10:41:31.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 10 monclient(hunting): dump: 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:31.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3982398111 0 0) 0x7fdf58005300 con 0x7fdf6c0324b0 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 >> v1:172.21.15.136:6789/0 conn(0x7fdf6c0d11e0 legacy=0x7fdf6c167060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.317+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 >> v1:172.21.15.117:6789/0 conn(0x7fdf6c038c60 legacy=0x7fdf6c15c620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 1 monclient: found mon.c 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c036340 con 0x7fdf6c0324b0 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf7ab3b640 5 monclient: authenticate success, global_id 4526 2024-03-29T10:41:31.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf7ab3b640 10 monclient: _renew_subs 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf7ab3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c035ef0 con 0x7fdf6c0324b0 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf7ab3b640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf7ab3b640 10 monclient: _renew_subs 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf7ab3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c15cf10 con 0x7fdf6c0324b0 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf7ab3b640 1 librados: init done 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf70001e00 con 0x7fdf6c0324b0 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.319+0000 7fdf6b7fe640 4 set_mon_vals no callback set 2024-03-29T10:41:31.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.320+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdf70001830 con 0x7fdf6c0324b0 2024-03-29T10:41:31.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.320+0000 7fdf6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:31.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.320+0000 7fdf6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:31.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.320+0000 7fdf6a7fc640 10 monclient: dump: 2024-03-29T10:41:31.309 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:31.309 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:31.309 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.320+0000 7fdf7ab3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.320+0000 7fdf7ab3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.320+0000 7fdf7ab3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.320+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c007b90 con 0x7fdf6c0324b0 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.321+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fdf7001a8b0 con 0x7fdf6c0324b0 2024-03-29T10:41:31.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.322+0000 7fdf79338640 10 monclient: get_auth_request con 0x7fdf5807f1d0 auth_method 0 2024-03-29T10:41:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.322+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fdf70054460 con 0x7fdf6c0324b0 2024-03-29T10:41:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.322+0000 7fdf6a7fc640 10 client.4526.objecter ms_dispatch 0x7fdf74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.322+0000 7fdf6a7fc640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.322+0000 7fdf6a7fc640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.322+0000 7fdf6a7fc640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T10:41:31.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.322+0000 7fdf6a7fc640 10 client.4526.objecter ms_handle_connect 0x7fdf5807f1d0 2024-03-29T10:41:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.332+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf70050700 con 0x7fdf6c0324b0 2024-03-29T10:41:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.332+0000 7fdf6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:31.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.332+0000 7fdf6a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:31.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.664+0000 7fdf7ab3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:31.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.664+0000 7fdf7ab3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:31.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.664+0000 7fdf7ab3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:41:31.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.664+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf6c004cd0 con 0x7fdf6c0324b0 2024-03-29T10:41:31.649 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:41:31.649 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:31.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.665+0000 7fdf6a7fc640 1 -- 172.21.15.117:0/4252903666 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fdf700508a0 con 0x7fdf6c0324b0 2024-03-29T10:41:31.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.665+0000 7fdf6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:31.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.665+0000 7fdf6a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:31.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.668+0000 7fdf7ab3b640 10 librados: watch_flush enter 2024-03-29T10:41:31.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.668+0000 7fdf7ab3b640 10 librados: watch_flush exit 2024-03-29T10:41:31.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.668+0000 7fdf7ab3b640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T10:41:31.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.668+0000 7fdf7ab3b640 10 client.4526.objecter successfully canceled tick 2024-03-29T10:41:31.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.668+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 >> v1:172.21.15.117:6816/376768171 conn(0x7fdf5807f1d0 legacy=0x7fdf58081690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:31.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.669+0000 7fdf7ab3b640 10 monclient: shutdown 2024-03-29T10:41:31.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.669+0000 7fdf7ab3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:31.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.669+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 >> v1:172.21.15.117:6790/0 conn(0x7fdf6c0324b0 legacy=0x7fdf6c0355e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:31.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.669+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 shutdown_connections 2024-03-29T10:41:31.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.669+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 >> 172.21.15.117:0/4252903666 conn(0x7fdf6c005c20 msgr2=0x7fdf6c006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:31.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.669+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 shutdown_connections 2024-03-29T10:41:31.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.669+0000 7fdf7ab3b640 1 -- 172.21.15.117:0/4252903666 wait complete. 2024-03-29T10:41:31.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:31.669+0000 7fdf7ab3b640 1 librados: shutdown 2024-03-29T10:41:31.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:38.479 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:38.621 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.638+0000 7f5f499c8640 10 monclient: get_monmap_and_config 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.638+0000 7f5f499c8640 10 monclient: build_initial_monmap 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.638+0000 7f5f499c8640 10 monclient: monmap: 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:41:38.639113+0000 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:41:38.639113+0000 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:38.622 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:41:38.623 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:41:38.623 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:38.624 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.639+0000 7f5f499c8640 1 Processor -- start 2024-03-29T10:41:38.624 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.639+0000 7f5f499c8640 1 -- start start 2024-03-29T10:41:38.624 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.639+0000 7f5f499c8640 10 monclient: init 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 10 monclient(hunting): picked mon.noname-c con 0x7f5f441457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 10 monclient(hunting): picked mon.noname-a con 0x7f5f4413f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 10 monclient(hunting): picked mon.noname-b con 0x7f5f440a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f440aaef0 con 0x7f5f4413f650 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f44145bd0 con 0x7f5f441457f0 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f44145eb0 con 0x7f5f440a9d70 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f499c8640 10 monclient(hunting): authenticate will time out at 1219.902886s 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f42ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5f441457f0 0x7f5f4413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:45276/0 (socket says 172.21.15.117:45276) 2024-03-29T10:41:38.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.640+0000 7f5f42ffd640 1 -- 172.21.15.117:0/126034395 learned_addr learned my addr 172.21.15.117:0/126034395 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.641+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119391619 0 0) 0x7f5f44145bd0 con 0x7f5f441457f0 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.641+0000 7f5f41ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.641+0000 7f5f41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.641+0000 7f5f41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.641+0000 7f5f41ffb640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.641+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f30003520 con 0x7f5f441457f0 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889837432 0 0) 0x7f5f440aaef0 con 0x7f5f4413f650 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:41:38.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f44145bd0 con 0x7f5f4413f650 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2146032815 0 0) 0x7f5f30003520 con 0x7f5f441457f0 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 >> v1:172.21.15.117:6790/0 conn(0x7f5f440a9d70 legacy=0x7f5f4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 >> v1:172.21.15.117:6789/0 conn(0x7f5f4413f650 legacy=0x7f5f440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 monclient: found mon.noname-c 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f44146340 con 0x7f5f441457f0 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: _finish_auth 0 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5f2c001690 con 0x7f5f441457f0 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:38.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: dump: 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:38.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f499c8640 5 monclient: authenticate success, global_id 4456 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f499c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f2c0017f0 con 0x7f5f441457f0 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 1 -- 172.21.15.117:0/126034395 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5f2c001c00 con 0x7f5f441457f0 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f41ffb640 10 monclient: dump: 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:38.632 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f499c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.642+0000 7f5f499c8640 4 set_mon_vals no callback set 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.643+0000 7f5f499c8640 10 monclient: shutdown 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.643+0000 7f5f499c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.643+0000 7f5f499c8640 1 -- 172.21.15.117:0/126034395 >> v1:172.21.15.136:6789/0 conn(0x7f5f441457f0 legacy=0x7f5f4413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.643+0000 7f5f499c8640 1 -- 172.21.15.117:0/126034395 shutdown_connections 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.643+0000 7f5f499c8640 1 -- 172.21.15.117:0/126034395 >> 172.21.15.117:0/126034395 conn(0x7f5f44066a00 msgr2=0x7f5f44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.643+0000 7f5f499c8640 1 -- 172.21.15.117:0/126034395 shutdown_connections 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.644+0000 7f5f499c8640 1 -- 172.21.15.117:0/126034395 wait complete. 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.644+0000 7f5f499c8640 10 monclient: build_initial_monmap 2024-03-29T10:41:38.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.644+0000 7f5f499c8640 10 monclient: monmap: 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.645+0000 7f5f499c8640 1 librados: starting msgr at 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.645+0000 7f5f499c8640 1 librados: starting objecter 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.645+0000 7f5f499c8640 1 Processor -- start 2024-03-29T10:41:38.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.645+0000 7f5f499c8640 1 -- start start 2024-03-29T10:41:38.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.645+0000 7f5f499c8640 1 librados: setting wanted keys 2024-03-29T10:41:38.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.645+0000 7f5f499c8640 1 librados: calling monclient init 2024-03-29T10:41:38.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.645+0000 7f5f499c8640 10 monclient: init 2024-03-29T10:41:38.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:38.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:38.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 10 monclient(hunting): picked mon.noname-a con 0x7f5f440a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4413f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 10 monclient(hunting): picked mon.noname-c con 0x7f5f441457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f44142ba0 con 0x7f5f440a9d70 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f42ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f5f440a9d70 0x7f5f44137e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:49874/0 (socket says 172.21.15.117:49874) 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f42ffd640 1 -- 172.21.15.117:0/2395285523 learned_addr learned my addr 172.21.15.117:0/2395285523 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f441ccf20 con 0x7f5f4413f650 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f441cd200 con 0x7f5f441457f0 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f499c8640 10 monclient(hunting): authenticate will time out at 1219.908786s 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f3bfff640 10 client.?.objecter ms_handle_connect 0x7f5f440a9d70 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.646+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241514524 0 0) 0x7f5f44142ba0 con 0x7f5f440a9d70 2024-03-29T10:41:38.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f24004eb0 con 0x7f5f440a9d70 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 client.?.objecter ms_handle_connect 0x7f5f441457f0 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1772973831 0 0) 0x7f5f24004eb0 con 0x7f5f440a9d70 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f44142ba0 con 0x7f5f440a9d70 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5f2c001c40 con 0x7f5f440a9d70 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient(hunting): dump: 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2877564957 0 0) 0x7f5f44142ba0 con 0x7f5f440a9d70 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 >> v1:172.21.15.117:6790/0 conn(0x7f5f441457f0 legacy=0x7f5f441e2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 >> v1:172.21.15.136:6789/0 conn(0x7f5f4413f650 legacy=0x7f5f441daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 1 monclient: found mon.a 2024-03-29T10:41:38.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f44142830 con 0x7f5f440a9d70 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient: _finish_auth 0 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f499c8640 5 monclient: authenticate success, global_id 4470 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f499c8640 10 monclient: _renew_subs 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f499c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f441cb340 con 0x7f5f440a9d70 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f499c8640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f499c8640 10 monclient: _renew_subs 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f499c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f44138750 con 0x7f5f440a9d70 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.647+0000 7f5f499c8640 1 librados: init done 2024-03-29T10:41:38.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.648+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f2c004a90 con 0x7f5f440a9d70 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.648+0000 7f5f3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.648+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5f2c004e60 con 0x7f5f440a9d70 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.648+0000 7f5f3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.648+0000 7f5f3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.648+0000 7f5f3bfff640 10 monclient: dump: 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.648+0000 7f5f417fa640 4 set_mon_vals no callback set 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.649+0000 7f5f499c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.649+0000 7f5f499c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.649+0000 7f5f499c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.649+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f28005080 con 0x7f5f440a9d70 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.651+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f5f2c004fc0 con 0x7f5f440a9d70 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.651+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f5f2c084bc0 con 0x7f5f440a9d70 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.651+0000 7f5f3bfff640 10 client.4470.objecter ms_dispatch 0x7f5f44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.651+0000 7f5f3bfff640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.651+0000 7f5f3bfff640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.651+0000 7f5f3bfff640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.651+0000 7f5f427fc640 10 monclient: get_auth_request con 0x7f5f2406dcb0 auth_method 0 2024-03-29T10:41:38.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.652+0000 7f5f3bfff640 10 client.4470.objecter ms_handle_connect 0x7f5f2406dcb0 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.659+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f2c0504a0 con 0x7f5f440a9d70 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.659+0000 7f5f3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.659+0000 7f5f3bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:38.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.979+0000 7f5f499c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:38.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.979+0000 7f5f499c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:38.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.979+0000 7f5f499c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:38.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.979+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f28005330 con 0x7f5f440a9d70 2024-03-29T10:41:38.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.979+0000 7f5f3bfff640 1 -- 172.21.15.117:0/2395285523 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f5f2c0523d0 con 0x7f5f440a9d70 2024-03-29T10:41:38.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.979+0000 7f5f3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:38.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.979+0000 7f5f3bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:38.963 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:41:38.963 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:38.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.982+0000 7f5f499c8640 10 librados: watch_flush enter 2024-03-29T10:41:38.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 10 librados: watch_flush exit 2024-03-29T10:41:38.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-29T10:41:38.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 10 client.4470.objecter successfully canceled tick 2024-03-29T10:41:38.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 >> v1:172.21.15.117:6816/376768171 conn(0x7f5f2406dcb0 legacy=0x7f5f24070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 10 monclient: shutdown 2024-03-29T10:41:38.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:38.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 >> v1:172.21.15.117:6789/0 conn(0x7f5f440a9d70 legacy=0x7f5f44137e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 shutdown_connections 2024-03-29T10:41:38.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 >> 172.21.15.117:0/2395285523 conn(0x7f5f44066a00 msgr2=0x7f5f440b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:38.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.983+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 shutdown_connections 2024-03-29T10:41:38.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.984+0000 7f5f499c8640 1 -- 172.21.15.117:0/2395285523 wait complete. 2024-03-29T10:41:38.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:38.984+0000 7f5f499c8640 1 librados: shutdown 2024-03-29T10:41:38.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:45.793 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:45.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.953+0000 7f841e78d640 10 monclient: get_monmap_and_config 2024-03-29T10:41:45.938 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.953+0000 7f841e78d640 10 monclient: build_initial_monmap 2024-03-29T10:41:45.938 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.953+0000 7f841e78d640 10 monclient: monmap: 2024-03-29T10:41:45.938 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:45.938 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:45.938 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:41:45.955326+0000 2024-03-29T10:41:45.938 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:41:45.955326+0000 2024-03-29T10:41:45.938 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:45.938 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:45.938 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:45.939 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:41:45.939 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:41:45.939 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:45.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.954+0000 7f841e78d640 1 Processor -- start 2024-03-29T10:41:45.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.955+0000 7f841e78d640 1 -- start start 2024-03-29T10:41:45.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.955+0000 7f841e78d640 10 monclient: init 2024-03-29T10:41:45.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:45.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:45.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 10 monclient(hunting): picked mon.noname-a con 0x7f8418149c50 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 10 monclient(hunting): picked mon.noname-b con 0x7f84180a9cc0 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 10 monclient(hunting): picked mon.noname-c con 0x7f8418143d00 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84180ab9c0 con 0x7f8418149c50 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84180abc10 con 0x7f8418143d00 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84180aad20 con 0x7f84180a9cc0 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f841e78d640 10 monclient(hunting): authenticate will time out at 1227.219688s 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f84177fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f84180a9cc0 0x7f84181435f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34918/0 (socket says 172.21.15.117:34918) 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.956+0000 7f84177fe640 1 -- 172.21.15.117:0/2621542047 learned_addr learned my addr 172.21.15.117:0/2621542047 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916902455 0 0) 0x7f84180ab9c0 con 0x7f8418149c50 2024-03-29T10:41:45.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8400003520 con 0x7f8418149c50 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875225740 0 0) 0x7f84180aad20 con 0x7f84180a9cc0 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:41:45.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.957+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84180ab9c0 con 0x7f84180a9cc0 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238740272 0 0) 0x7f84180abc10 con 0x7f8418143d00 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84180aad20 con 0x7f8418143d00 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 942046355 0 0) 0x7f84180aad20 con 0x7f8418143d00 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 >> v1:172.21.15.117:6790/0 conn(0x7f84180a9cc0 legacy=0x7f84181435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 >> v1:172.21.15.117:6789/0 conn(0x7f8418149c50 legacy=0x7f84180ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:45.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 1 monclient: found mon.noname-c 2024-03-29T10:41:45.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:45.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:41:45.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84180aa980 con 0x7f8418143d00 2024-03-29T10:41:45.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:45.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:45.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:45.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f841e78d640 5 monclient: authenticate success, global_id 4459 2024-03-29T10:41:45.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f841e78d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:45.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f84100018e0 con 0x7f8418143d00 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.958+0000 7f8416ffd640 10 monclient: dump: 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:45.952 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.959+0000 7f841e78d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.959+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8410001b00 con 0x7f8418143d00 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.959+0000 7f8416ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.959+0000 7f8416ffd640 1 -- 172.21.15.117:0/2621542047 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8410001e10 con 0x7f8418143d00 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.959+0000 7f8416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.959+0000 7f8416ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.959+0000 7f8416ffd640 10 monclient: dump: 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:45.953 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.959+0000 7f841e78d640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.959+0000 7f841e78d640 4 set_mon_vals no callback set 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.961+0000 7f841e78d640 10 monclient: shutdown 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.961+0000 7f841e78d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.961+0000 7f841e78d640 1 -- 172.21.15.117:0/2621542047 >> v1:172.21.15.136:6789/0 conn(0x7f8418143d00 legacy=0x7f8418151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.961+0000 7f841e78d640 1 -- 172.21.15.117:0/2621542047 shutdown_connections 2024-03-29T10:41:45.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.961+0000 7f841e78d640 1 -- 172.21.15.117:0/2621542047 >> 172.21.15.117:0/2621542047 conn(0x7f84180b3c50 msgr2=0x7f84180b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:45.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.961+0000 7f841e78d640 1 -- 172.21.15.117:0/2621542047 shutdown_connections 2024-03-29T10:41:45.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.961+0000 7f841e78d640 1 -- 172.21.15.117:0/2621542047 wait complete. 2024-03-29T10:41:45.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.962+0000 7f841e78d640 10 monclient: build_initial_monmap 2024-03-29T10:41:45.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.962+0000 7f841e78d640 10 monclient: monmap: 2024-03-29T10:41:45.958 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:45.958 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.962+0000 7f841e78d640 1 librados: starting msgr at 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.962+0000 7f841e78d640 1 librados: starting objecter 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.962+0000 7f841e78d640 1 Processor -- start 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.963+0000 7f841e78d640 1 -- start start 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.963+0000 7f841e78d640 1 librados: setting wanted keys 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.963+0000 7f841e78d640 1 librados: calling monclient init 2024-03-29T10:41:45.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.963+0000 7f841e78d640 10 monclient: init 2024-03-29T10:41:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 10 monclient(hunting): picked mon.noname-c con 0x7f84180a9cc0 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 10 monclient(hunting): picked mon.noname-a con 0x7f8418143d00 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 10 monclient(hunting): picked mon.noname-b con 0x7f8418149c50 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84181df630 con 0x7f8418143d00 2024-03-29T10:41:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8418147020 con 0x7f8418149c50 2024-03-29T10:41:45.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84181df900 con 0x7f84180a9cc0 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f841e78d640 10 monclient(hunting): authenticate will time out at 1227.227457s 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f8417fff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f84180a9cc0 0x7f84181def20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:35266/0 (socket says 172.21.15.117:35266) 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.964+0000 7f8417fff640 1 -- 172.21.15.117:0/1106402298 learned_addr learned my addr 172.21.15.117:0/1106402298 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 client.?.objecter ms_handle_connect 0x7f84180a9cc0 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 client.?.objecter ms_handle_connect 0x7f8418149c50 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324105971 0 0) 0x7f8418147020 con 0x7f8418149c50 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:41:45.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8404004eb0 con 0x7f8418149c50 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 client.?.objecter ms_handle_connect 0x7f8418143d00 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620362428 0 0) 0x7f84181df900 con 0x7f84180a9cc0 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.965+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8418147020 con 0x7f84180a9cc0 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2897351006 0 0) 0x7f8404004eb0 con 0x7f8418149c50 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84181df900 con 0x7f8418149c50 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f84100017a0 con 0x7f8418149c50 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:45.968 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1795595547 0 0) 0x7f8418147020 con 0x7f84180a9cc0 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8404004eb0 con 0x7f84180a9cc0 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023223948 0 0) 0x7f84181df630 con 0x7f8418143d00 2024-03-29T10:41:45.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8418147020 con 0x7f8418143d00 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.966+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1618713458 0 0) 0x7f84181df900 con 0x7f8418149c50 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 >> v1:172.21.15.117:6790/0 conn(0x7f84180a9cc0 legacy=0x7f84181def20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 >> v1:172.21.15.117:6789/0 conn(0x7f8418143d00 legacy=0x7f8418144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 1 monclient: found mon.b 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8418072c70 con 0x7f8418149c50 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 10 monclient: _finish_auth 0 2024-03-29T10:41:45.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f8414ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f841e78d640 5 monclient: authenticate success, global_id 4462 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f841e78d640 10 monclient: _renew_subs 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f841e78d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84181453b0 con 0x7f8418149c50 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f841e78d640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f841e78d640 10 monclient: _renew_subs 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f841e78d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84180723b0 con 0x7f8418149c50 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.967+0000 7f841e78d640 1 librados: init done 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.968+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8410003c20 con 0x7f8418149c50 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.968+0000 7f8414ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.968+0000 7f84167fc640 4 set_mon_vals no callback set 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.968+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8410003f30 con 0x7f8418149c50 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.968+0000 7f8414ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:45.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.968+0000 7f8414ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.968+0000 7f8414ff9640 10 monclient: dump: 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.969+0000 7f841e78d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.969+0000 7f841e78d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.969+0000 7f841e78d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.969+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f8005080 con 0x7f8418149c50 2024-03-29T10:41:45.972 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.970+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f8410019f00 con 0x7f8418149c50 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.971+0000 7f8417fff640 10 monclient: get_auth_request con 0x7f840406e0e0 auth_method 0 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.971+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f8410083c20 con 0x7f8418149c50 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.971+0000 7f8414ff9640 10 client.4462.objecter ms_dispatch 0x7f8418000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.971+0000 7f8414ff9640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.971+0000 7f8414ff9640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.971+0000 7f8414ff9640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.972+0000 7f8414ff9640 10 client.4462.objecter ms_handle_connect 0x7f840406e0e0 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.977+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f841004f500 con 0x7f8418149c50 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.977+0000 7f8414ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:45.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:45.977+0000 7f8414ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:46.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.338+0000 7f841e78d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:46.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.338+0000 7f841e78d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:46.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.338+0000 7f841e78d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:41:46.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.338+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83f8005330 con 0x7f8418149c50 2024-03-29T10:41:46.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.339+0000 7f8414ff9640 1 -- 172.21.15.117:0/1106402298 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f8410051430 con 0x7f8418149c50 2024-03-29T10:41:46.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.339+0000 7f8414ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:46.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.339+0000 7f8414ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:46.324 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:41:46.324 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:46.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.343+0000 7f841e78d640 10 librados: watch_flush enter 2024-03-29T10:41:46.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.343+0000 7f841e78d640 10 librados: watch_flush exit 2024-03-29T10:41:46.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.343+0000 7f841e78d640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-29T10:41:46.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.343+0000 7f841e78d640 10 client.4462.objecter successfully canceled tick 2024-03-29T10:41:46.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.343+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 >> v1:172.21.15.117:6816/376768171 conn(0x7f840406e0e0 legacy=0x7f84040705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:46.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.343+0000 7f841e78d640 10 monclient: shutdown 2024-03-29T10:41:46.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.343+0000 7f841e78d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:46.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.343+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 >> v1:172.21.15.136:6789/0 conn(0x7f8418149c50 legacy=0x7f8418071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:46.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.344+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 shutdown_connections 2024-03-29T10:41:46.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.344+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 >> 172.21.15.117:0/1106402298 conn(0x7f84180b3c50 msgr2=0x7f84180b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:46.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.344+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 shutdown_connections 2024-03-29T10:41:46.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.344+0000 7f841e78d640 1 -- 172.21.15.117:0/1106402298 wait complete. 2024-03-29T10:41:46.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:46.344+0000 7f841e78d640 1 librados: shutdown 2024-03-29T10:41:46.343 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:53.150 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.318+0000 7fdbe61e6640 10 monclient: get_monmap_and_config 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.318+0000 7fdbe61e6640 10 monclient: build_initial_monmap 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.318+0000 7fdbe61e6640 10 monclient: monmap: 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:41:53.319402+0000 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:41:53.319402+0000 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:53.307 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.319+0000 7fdbe61e6640 1 Processor -- start 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.319+0000 7fdbe61e6640 1 -- start start 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.319+0000 7fdbe61e6640 10 monclient: init 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe00ad6c0 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe00af6c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe014fdc0 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00a94e0 con 0x7fdbe00af6c0 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00af3a0 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00afb60 con 0x7fdbe014fdc0 2024-03-29T10:41:53.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.320+0000 7fdbe61e6640 10 monclient(hunting): authenticate will time out at 1234.582647s 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.322+0000 7fdbdeffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fdbe00af6c0 0x7fdbe0152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:59204/0 (socket says 172.21.15.117:59204) 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.322+0000 7fdbdeffd640 1 -- 172.21.15.117:0/2623492022 learned_addr learned my addr 172.21.15.117:0/2623492022 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.322+0000 7fdbde7fc640 1 -- 172.21.15.117:0/2623492022 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664257294 0 0) 0x7fdbe00a94e0 con 0x7fdbe00af6c0 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.322+0000 7fdbde7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.322+0000 7fdbde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.322+0000 7fdbde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.322+0000 7fdbde7fc640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.322+0000 7fdbde7fc640 1 -- 172.21.15.117:0/2623492022 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc8003520 con 0x7fdbe00af6c0 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 1 -- 172.21.15.117:0/2623492022 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2728623250 0 0) 0x7fdbc8003520 con 0x7fdbe00af6c0 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 1 -- 172.21.15.117:0/2623492022 >> v1:172.21.15.117:6790/0 conn(0x7fdbe014fdc0 legacy=0x7fdbe015a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 1 -- 172.21.15.117:0/2623492022 >> v1:172.21.15.136:6789/0 conn(0x7fdbe00ad6c0 legacy=0x7fdbe014f6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:53.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 1 monclient: found mon.noname-a 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 1 -- 172.21.15.117:0/2623492022 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe00aab80 con 0x7fdbe00af6c0 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbe61e6640 5 monclient: authenticate success, global_id 4479 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbe61e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 1 -- 172.21.15.117:0/2623492022 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdbd40012f0 con 0x7fdbe00af6c0 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 10 monclient: dump: 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:53.310 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbe61e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 1 -- 172.21.15.117:0/2623492022 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd40014c0 con 0x7fdbe00af6c0 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbe61e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:53.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.323+0000 7fdbe61e6640 4 set_mon_vals no callback set 2024-03-29T10:41:53.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.324+0000 7fdbde7fc640 1 -- 172.21.15.117:0/2623492022 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdbd4001910 con 0x7fdbe00af6c0 2024-03-29T10:41:53.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.324+0000 7fdbde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:53.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.324+0000 7fdbde7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:53.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.324+0000 7fdbde7fc640 10 monclient: dump: 2024-03-29T10:41:53.312 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:53.312 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:53.312 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:53.312 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:53.312 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.324+0000 7fdbe61e6640 10 monclient: shutdown 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.324+0000 7fdbe61e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.324+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2623492022 >> v1:172.21.15.117:6789/0 conn(0x7fdbe00af6c0 legacy=0x7fdbe0152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.325+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2623492022 shutdown_connections 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.325+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2623492022 >> 172.21.15.117:0/2623492022 conn(0x7fdbe00b35e0 msgr2=0x7fdbe00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.325+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2623492022 shutdown_connections 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.325+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2623492022 wait complete. 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.326+0000 7fdbe61e6640 10 monclient: build_initial_monmap 2024-03-29T10:41:53.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.326+0000 7fdbe61e6640 10 monclient: monmap: 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.326+0000 7fdbe61e6640 1 librados: starting msgr at 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.326+0000 7fdbe61e6640 1 librados: starting objecter 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.326+0000 7fdbe61e6640 1 Processor -- start 2024-03-29T10:41:53.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.326+0000 7fdbe61e6640 1 -- start start 2024-03-29T10:41:53.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.326+0000 7fdbe61e6640 1 librados: setting wanted keys 2024-03-29T10:41:53.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.326+0000 7fdbe61e6640 1 librados: calling monclient init 2024-03-29T10:41:53.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.326+0000 7fdbe61e6640 10 monclient: init 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe00ad6c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe00af6c0 addr v1:172.21.15.117:6790/0 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe014fdc0 addr v1:172.21.15.136:6789/0 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe006fc90 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe008b050 con 0x7fdbe014fdc0 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe0151ec0 con 0x7fdbe00af6c0 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.327+0000 7fdbe61e6640 10 monclient(hunting): authenticate will time out at 1234.589786s 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbdf7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fdbe00ad6c0 0x7fdbe01e8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:60330/0 (socket says 172.21.15.117:60330) 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbdf7fe640 1 -- 172.21.15.117:0/2250989266 learned_addr learned my addr 172.21.15.117:0/2250989266 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbc7fff640 10 client.?.objecter ms_handle_connect 0x7fdbe00ad6c0 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060706049 0 0) 0x7fdbe006fc90 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbc7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbc7fff640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.330+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbcc004eb0 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.331+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077996105 0 0) 0x7fdbcc004eb0 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.331+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbe006fc90 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.331+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdbd8001bf0 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.331+0000 7fdbc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.331+0000 7fdbc7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.331+0000 7fdbc7fff640 10 monclient(hunting): dump: 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.331+0000 7fdbc7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.331+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1767197826 0 0) 0x7fdbe006fc90 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.331+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 >> v1:172.21.15.117:6790/0 conn(0x7fdbe00af6c0 legacy=0x7fdbe008a230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.332+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 >> v1:172.21.15.136:6789/0 conn(0x7fdbe014fdc0 legacy=0x7fdbe008a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.332+0000 7fdbc7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.332+0000 7fdbc7fff640 1 monclient: found mon.a 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbc7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe01515b0 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbc7fff640 10 monclient: _finish_auth 0 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbc7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbe61e6640 5 monclient: authenticate success, global_id 4482 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbe61e6640 10 monclient: _renew_subs 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbe61e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe006ec80 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbe61e6640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbe61e6640 10 monclient: _renew_subs 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbe61e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe006e980 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.333+0000 7fdbe61e6640 1 librados: init done 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.334+0000 7fdbc7fff640 10 client.4482.objecter ms_handle_connect 0x7fdbe014fdc0 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.334+0000 7fdbc7fff640 10 client.4482.objecter resend_mon_ops 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.334+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd8001d50 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.334+0000 7fdbc7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.334+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdbd8004e60 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.334+0000 7fdbc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.334+0000 7fdbc7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.334+0000 7fdbc7fff640 10 monclient: dump: 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:41:53.323 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:41:53.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.335+0000 7fdbddffb640 4 set_mon_vals no callback set 2024-03-29T10:41:53.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.335+0000 7fdbe61e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:53.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.336+0000 7fdbe61e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.336+0000 7fdbe61e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.336+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc0005080 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.338+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fdbd8004fc0 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.339+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fdbd8084e70 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.339+0000 7fdbc7fff640 10 client.4482.objecter ms_dispatch 0x7fdbe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.339+0000 7fdbc7fff640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.339+0000 7fdbc7fff640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.339+0000 7fdbc7fff640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.339+0000 7fdbdeffd640 10 monclient: get_auth_request con 0x7fdbcc06db60 auth_method 0 2024-03-29T10:41:53.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.339+0000 7fdbc7fff640 10 client.4482.objecter ms_handle_connect 0x7fdbcc06db60 2024-03-29T10:41:53.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.347+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbd8050580 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.347+0000 7fdbc7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:53.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.347+0000 7fdbc7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:53.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.700+0000 7fdbe61e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:53.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.700+0000 7fdbe61e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:53.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.700+0000 7fdbe61e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:41:53.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.700+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbc0005330 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.701+0000 7fdbc7fff640 1 -- 172.21.15.117:0/2250989266 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fdbd8052680 con 0x7fdbe00ad6c0 2024-03-29T10:41:53.684 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:41:53.685 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:53.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.701+0000 7fdbc7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:53.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.701+0000 7fdbc7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:53.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.704+0000 7fdbe61e6640 10 librados: watch_flush enter 2024-03-29T10:41:53.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 10 librados: watch_flush exit 2024-03-29T10:41:53.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T10:41:53.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 10 client.4482.objecter successfully canceled tick 2024-03-29T10:41:53.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 >> v1:172.21.15.117:6816/376768171 conn(0x7fdbcc06db60 legacy=0x7fdbcc070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:53.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 10 monclient: shutdown 2024-03-29T10:41:53.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:53.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 >> v1:172.21.15.117:6789/0 conn(0x7fdbe00ad6c0 legacy=0x7fdbe01e8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:53.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 shutdown_connections 2024-03-29T10:41:53.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 >> 172.21.15.117:0/2250989266 conn(0x7fdbe00b35e0 msgr2=0x7fdbe00a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:53.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 shutdown_connections 2024-03-29T10:41:53.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 1 -- 172.21.15.117:0/2250989266 wait complete. 2024-03-29T10:41:53.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:41:53.705+0000 7fdbe61e6640 1 librados: shutdown 2024-03-29T10:41:53.703 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:00.510 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:00.661 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.676+0000 7fe765284640 10 monclient: get_monmap_and_config 2024-03-29T10:42:00.661 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.676+0000 7fe765284640 10 monclient: build_initial_monmap 2024-03-29T10:42:00.661 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.677+0000 7fe765284640 10 monclient: monmap: 2024-03-29T10:42:00.661 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:00.661 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:00.662 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:42:00.677915+0000 2024-03-29T10:42:00.662 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:42:00.677915+0000 2024-03-29T10:42:00.662 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:00.662 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:00.662 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:00.662 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:42:00.662 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:42:00.662 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:00.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.678+0000 7fe765284640 1 Processor -- start 2024-03-29T10:42:00.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.678+0000 7fe765284640 1 -- start start 2024-03-29T10:42:00.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.678+0000 7fe765284640 10 monclient: init 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.679+0000 7fe765284640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.679+0000 7fe765284640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.679+0000 7fe765284640 10 monclient(hunting): picked mon.noname-a con 0x7fe7600b4100 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.679+0000 7fe765284640 10 monclient(hunting): picked mon.noname-c con 0x7fe76013ee90 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.679+0000 7fe765284640 10 monclient(hunting): picked mon.noname-b con 0x7fe7600af900 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.679+0000 7fe75effd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe7600b4100 0x7fe7600adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:32918/0 (socket says 172.21.15.117:32918) 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.679+0000 7fe75effd640 1 -- 172.21.15.117:0/552733806 learned_addr learned my addr 172.21.15.117:0/552733806 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe765284640 1 -- 172.21.15.117:0/552733806 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600afd10 con 0x7fe7600b4100 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe765284640 1 -- 172.21.15.117:0/552733806 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600af6f0 con 0x7fe76013ee90 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe765284640 1 -- 172.21.15.117:0/552733806 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600aac80 con 0x7fe7600af900 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe765284640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:00.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe765284640 10 monclient(hunting): authenticate will time out at 1241.942637s 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114809474 0 0) 0x7fe7600afd10 con 0x7fe7600b4100 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe75e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe75e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe75e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe75e7fc640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c005730 con 0x7fe7600b4100 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851312108 0 0) 0x7fe7600aac80 con 0x7fe7600af900 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe75e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.680+0000 7fe75e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7600afd10 con 0x7fe7600af900 2024-03-29T10:42:00.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163336633 0 0) 0x7fe7600af6f0 con 0x7fe76013ee90 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7600aac80 con 0x7fe76013ee90 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 353034240 0 0) 0x7fe74c005730 con 0x7fe7600b4100 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 >> v1:172.21.15.117:6790/0 conn(0x7fe7600af900 legacy=0x7fe76014ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 >> v1:172.21.15.136:6789/0 conn(0x7fe76013ee90 legacy=0x7fe76013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 1 monclient: found mon.noname-a 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600aaf60 con 0x7fe7600b4100 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:00.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe750001610 con 0x7fe7600b4100 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe75e7fc640 10 monclient: dump: 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:00.668 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:00.669 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:00.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe765284640 5 monclient: authenticate success, global_id 4485 2024-03-29T10:42:00.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.681+0000 7fe765284640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:00.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.682+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe750001770 con 0x7fe7600b4100 2024-03-29T10:42:00.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.682+0000 7fe75e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:00.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.682+0000 7fe75e7fc640 1 -- 172.21.15.117:0/552733806 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe750001bc0 con 0x7fe7600b4100 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.682+0000 7fe75e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.682+0000 7fe75e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.682+0000 7fe75e7fc640 10 monclient: dump: 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:00.676 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.682+0000 7fe765284640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.682+0000 7fe765284640 4 set_mon_vals no callback set 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.683+0000 7fe765284640 10 monclient: shutdown 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.683+0000 7fe765284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.683+0000 7fe765284640 1 -- 172.21.15.117:0/552733806 >> v1:172.21.15.117:6789/0 conn(0x7fe7600b4100 legacy=0x7fe7600adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.683+0000 7fe765284640 1 -- 172.21.15.117:0/552733806 shutdown_connections 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.683+0000 7fe765284640 1 -- 172.21.15.117:0/552733806 >> 172.21.15.117:0/552733806 conn(0x7fe7600b35e0 msgr2=0x7fe7600b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.683+0000 7fe765284640 1 -- 172.21.15.117:0/552733806 shutdown_connections 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.683+0000 7fe765284640 1 -- 172.21.15.117:0/552733806 wait complete. 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.684+0000 7fe765284640 10 monclient: build_initial_monmap 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.685+0000 7fe765284640 10 monclient: monmap: 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:00.677 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.685+0000 7fe765284640 1 librados: starting msgr at 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.685+0000 7fe765284640 1 librados: starting objecter 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.685+0000 7fe765284640 1 Processor -- start 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.686+0000 7fe765284640 1 -- start start 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.686+0000 7fe765284640 1 librados: setting wanted keys 2024-03-29T10:42:00.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.686+0000 7fe765284640 1 librados: calling monclient init 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.686+0000 7fe765284640 10 monclient: init 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 10 monclient(hunting): picked mon.noname-c con 0x7fe7600af900 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe75effd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe7600af900 0x7fe760138e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:39506/0 (socket says 172.21.15.117:39506) 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe75effd640 1 -- 172.21.15.117:0/2703807425 learned_addr learned my addr 172.21.15.117:0/2703807425 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 10 monclient(hunting): picked mon.noname-a con 0x7fe7600b4100 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe7577fe640 10 client.?.objecter ms_handle_connect 0x7fe7600af900 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe7577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 10 monclient(hunting): picked mon.noname-b con 0x7fe76013ee90 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe760138130 con 0x7fe7600b4100 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe7577fe640 10 client.?.objecter ms_handle_connect 0x7fe7600b4100 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe7577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76013a2f0 con 0x7fe76013ee90 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7601d34b0 con 0x7fe7600af900 2024-03-29T10:42:00.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.687+0000 7fe765284640 10 monclient(hunting): authenticate will time out at 1241.949959s 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715793766 0 0) 0x7fe760138130 con 0x7fe7600b4100 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe740004eb0 con 0x7fe7600b4100 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726721095 0 0) 0x7fe7601d34b0 con 0x7fe7600af900 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:42:00.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe760138130 con 0x7fe7600af900 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2374020170 0 0) 0x7fe740004eb0 con 0x7fe7600b4100 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.688+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7601d34b0 con 0x7fe7600b4100 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe748001850 con 0x7fe7600b4100 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): dump: 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:00.681 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 client.?.objecter ms_handle_connect 0x7fe76013ee90 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417522724 0 0) 0x7fe76013a2f0 con 0x7fe76013ee90 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe740004eb0 con 0x7fe76013ee90 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1143388743 0 0) 0x7fe760138130 con 0x7fe7600af900 2024-03-29T10:42:00.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe76013a2f0 con 0x7fe7600af900 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe750001ac0 con 0x7fe7600af900 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): dump: 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:00.683 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 666427015 0 0) 0x7fe740004eb0 con 0x7fe76013ee90 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe760138130 con 0x7fe76013ee90 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe758001cb0 con 0x7fe76013ee90 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient(hunting): dump: 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:00.684 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3926763662 0 0) 0x7fe7601d34b0 con 0x7fe7600b4100 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 >> v1:172.21.15.117:6790/0 conn(0x7fe7600af900 legacy=0x7fe760138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 >> v1:172.21.15.136:6789/0 conn(0x7fe76013ee90 legacy=0x7fe760141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 monclient: found mon.a 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe760134570 con 0x7fe7600b4100 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe7577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:00.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.689+0000 7fe765284640 5 monclient: authenticate success, global_id 4488 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe765284640 10 monclient: _renew_subs 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe765284640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe760134820 con 0x7fe7600b4100 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe748001d60 con 0x7fe7600b4100 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe7577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe7480037b0 con 0x7fe7600b4100 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe7577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe7577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe7577fe640 10 monclient: dump: 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:00.686 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe75dffb640 4 set_mon_vals no callback set 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.690+0000 7fe765284640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.691+0000 7fe765284640 10 monclient: _renew_subs 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.691+0000 7fe765284640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.691+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7601424a0 con 0x7fe7600b4100 2024-03-29T10:42:00.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.691+0000 7fe765284640 1 librados: init done 2024-03-29T10:42:00.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.692+0000 7fe765284640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:00.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.692+0000 7fe765284640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:00.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.692+0000 7fe765284640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:00.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.692+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7601343c0 con 0x7fe7600b4100 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.693+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fe748019780 con 0x7fe7600b4100 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.693+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fe748083750 con 0x7fe7600b4100 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.693+0000 7fe7577fe640 10 client.4488.objecter ms_dispatch 0x7fe760000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.693+0000 7fe7577fe640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.693+0000 7fe7577fe640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.693+0000 7fe7577fe640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.693+0000 7fe75effd640 10 monclient: get_auth_request con 0x7fe74006e060 auth_method 0 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.694+0000 7fe7577fe640 10 client.4488.objecter ms_handle_connect 0x7fe74006e060 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.705+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe74804ee60 con 0x7fe7600b4100 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.705+0000 7fe7577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:00.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:00.705+0000 7fe7577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:01.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.038+0000 7fe765284640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:01.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.038+0000 7fe765284640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.038+0000 7fe765284640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.038+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7601382e0 con 0x7fe7600b4100 2024-03-29T10:42:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.039+0000 7fe7577fe640 1 -- 172.21.15.117:0/2703807425 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fe7601382e0 con 0x7fe7600b4100 2024-03-29T10:42:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.039+0000 7fe7577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.039+0000 7fe7577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:01.023 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:42:01.023 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 10 librados: watch_flush enter 2024-03-29T10:42:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 10 librados: watch_flush exit 2024-03-29T10:42:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T10:42:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 10 client.4488.objecter successfully canceled tick 2024-03-29T10:42:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 >> v1:172.21.15.117:6816/376768171 conn(0x7fe74006e060 legacy=0x7fe740070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 10 monclient: shutdown 2024-03-29T10:42:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:01.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 >> v1:172.21.15.117:6789/0 conn(0x7fe7600b4100 legacy=0x7fe7601d2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:01.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 shutdown_connections 2024-03-29T10:42:01.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.042+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 >> 172.21.15.117:0/2703807425 conn(0x7fe7600b35e0 msgr2=0x7fe7601c9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:01.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.043+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 shutdown_connections 2024-03-29T10:42:01.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.043+0000 7fe765284640 1 -- 172.21.15.117:0/2703807425 wait complete. 2024-03-29T10:42:01.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:01.043+0000 7fe765284640 1 librados: shutdown 2024-03-29T10:42:01.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:07.849 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:08.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.019+0000 7f51849a3640 10 monclient: get_monmap_and_config 2024-03-29T10:42:08.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.019+0000 7f51849a3640 10 monclient: build_initial_monmap 2024-03-29T10:42:08.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.019+0000 7f51849a3640 10 monclient: monmap: 2024-03-29T10:42:08.003 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:08.003 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:08.003 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:42:08.020851+0000 2024-03-29T10:42:08.003 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:42:08.020851+0000 2024-03-29T10:42:08.003 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:08.004 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:08.004 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:08.004 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:42:08.004 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:42:08.004 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.022+0000 7f51849a3640 1 Processor -- start 2024-03-29T10:42:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.022+0000 7f51849a3640 1 -- start start 2024-03-29T10:42:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.022+0000 7f51849a3640 10 monclient: init 2024-03-29T10:42:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 10 monclient(hunting): picked mon.noname-a con 0x7f51800b0860 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 10 monclient(hunting): picked mon.noname-c con 0x7f518013ef30 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 10 monclient(hunting): picked mon.noname-b con 0x7f51800af8e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:08.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51800b0c40 con 0x7f51800b0860 2024-03-29T10:42:08.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51800aed10 con 0x7f518013ef30 2024-03-29T10:42:08.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51800afd50 con 0x7f51800af8e0 2024-03-29T10:42:08.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:08.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f51849a3640 10 monclient(hunting): authenticate will time out at 1249.285368s 2024-03-29T10:42:08.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f517dd9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f518013ef30 0x7f51800af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:40150/0 (socket says 172.21.15.117:40150) 2024-03-29T10:42:08.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.023+0000 7f517dd9b640 1 -- 172.21.15.117:0/1122678013 learned_addr learned my addr 172.21.15.117:0/1122678013 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:08.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.026+0000 7f517d59a640 1 -- 172.21.15.117:0/1122678013 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894817141 0 0) 0x7f51800afd50 con 0x7f51800af8e0 2024-03-29T10:42:08.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.026+0000 7f517d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:08.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.026+0000 7f517d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:08.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.026+0000 7f517d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:08.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.026+0000 7f517d59a640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:42:08.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.026+0000 7f517d59a640 1 -- 172.21.15.117:0/1122678013 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5168003520 con 0x7f51800af8e0 2024-03-29T10:42:08.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.027+0000 7f517d59a640 1 -- 172.21.15.117:0/1122678013 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4219045826 0 0) 0x7f5168003520 con 0x7f51800af8e0 2024-03-29T10:42:08.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.027+0000 7f517d59a640 1 -- 172.21.15.117:0/1122678013 >> v1:172.21.15.136:6789/0 conn(0x7f518013ef30 legacy=0x7f51800af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:08.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.027+0000 7f517d59a640 1 -- 172.21.15.117:0/1122678013 >> v1:172.21.15.117:6789/0 conn(0x7f51800b0860 legacy=0x7f518013e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 1 monclient: found mon.noname-b 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 1 -- 172.21.15.117:0/1122678013 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51800ab1b0 con 0x7f51800af8e0 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 10 monclient: _finish_auth 0 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 1 -- 172.21.15.117:0/1122678013 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5174001670 con 0x7f51800af8e0 2024-03-29T10:42:08.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f517d59a640 10 monclient: dump: 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:08.014 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f51849a3640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.028+0000 7f51849a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.029+0000 7f517d59a640 1 -- 172.21.15.117:0/1122678013 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5174001900 con 0x7f51800af8e0 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.029+0000 7f517d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.029+0000 7f51849a3640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.029+0000 7f51849a3640 4 set_mon_vals no callback set 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.029+0000 7f517d59a640 1 -- 172.21.15.117:0/1122678013 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5174001cd0 con 0x7f51800af8e0 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.029+0000 7f517d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.029+0000 7f517d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.029+0000 7f517d59a640 10 monclient: dump: 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:08.015 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:08.016 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:08.016 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:08.016 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:08.016 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:08.016 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:08.016 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:08.016 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:08.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.030+0000 7f51849a3640 10 monclient: shutdown 2024-03-29T10:42:08.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.030+0000 7f51849a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:08.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.030+0000 7f51849a3640 1 -- 172.21.15.117:0/1122678013 >> v1:172.21.15.117:6790/0 conn(0x7f51800af8e0 legacy=0x7f5180149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:08.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.030+0000 7f51849a3640 1 -- 172.21.15.117:0/1122678013 shutdown_connections 2024-03-29T10:42:08.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.030+0000 7f51849a3640 1 -- 172.21.15.117:0/1122678013 >> 172.21.15.117:0/1122678013 conn(0x7f518006b660 msgr2=0x7f518006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:08.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.032+0000 7f51849a3640 1 -- 172.21.15.117:0/1122678013 shutdown_connections 2024-03-29T10:42:08.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.033+0000 7f51849a3640 1 -- 172.21.15.117:0/1122678013 wait complete. 2024-03-29T10:42:08.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.035+0000 7f51849a3640 10 monclient: build_initial_monmap 2024-03-29T10:42:08.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.035+0000 7f51849a3640 10 monclient: monmap: 2024-03-29T10:42:08.019 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:08.019 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:08.019 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:42:08.019 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:42:08.019 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.035+0000 7f51849a3640 1 librados: starting msgr at 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.035+0000 7f51849a3640 1 librados: starting objecter 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.035+0000 7f51849a3640 1 Processor -- start 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.035+0000 7f51849a3640 1 -- start start 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.035+0000 7f51849a3640 1 librados: setting wanted keys 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.035+0000 7f51849a3640 1 librados: calling monclient init 2024-03-29T10:42:08.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.035+0000 7f51849a3640 10 monclient: init 2024-03-29T10:42:08.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.036+0000 7f51849a3640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:08.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.036+0000 7f51849a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:08.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.036+0000 7f51849a3640 10 monclient(hunting): picked mon.noname-c con 0x7f51800af8e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:08.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.036+0000 7f51849a3640 10 monclient(hunting): picked mon.noname-a con 0x7f51800b0860 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:08.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.040+0000 7f51849a3640 10 monclient(hunting): picked mon.noname-b con 0x7f518013ef30 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:08.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.040+0000 7f51849a3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51801d65d0 con 0x7f51800b0860 2024-03-29T10:42:08.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.040+0000 7f51849a3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51800b2750 con 0x7f518013ef30 2024-03-29T10:42:08.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.040+0000 7f517ed9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f518013ef30 0x7f51801de820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:42558/0 (socket says 172.21.15.117:42558) 2024-03-29T10:42:08.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.040+0000 7f517ed9d640 1 -- 172.21.15.117:0/2626230897 learned_addr learned my addr 172.21.15.117:0/2626230897 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:08.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.040+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51801d71c0 con 0x7f51800af8e0 2024-03-29T10:42:08.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.040+0000 7f51849a3640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:08.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.040+0000 7f51849a3640 10 monclient(hunting): authenticate will time out at 1249.302745s 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.041+0000 7f516effd640 10 client.?.objecter ms_handle_connect 0x7f51800b0860 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.041+0000 7f516effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.041+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961059802 0 0) 0x7f51801d65d0 con 0x7f51800b0860 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.041+0000 7f516effd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.041+0000 7f516effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.041+0000 7f516effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.041+0000 7f516effd640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.041+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5160004eb0 con 0x7f51800b0860 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 client.?.objecter ms_handle_connect 0x7f51800af8e0 2024-03-29T10:42:08.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:08.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266147083 0 0) 0x7f51801d71c0 con 0x7f51800af8e0 2024-03-29T10:42:08.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:08.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:08.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:08.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:42:08.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51801d65d0 con 0x7f51800af8e0 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2589180048 0 0) 0x7f51801d65d0 con 0x7f51800af8e0 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5160005490 con 0x7f51800af8e0 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5178001b90 con 0x7f51800af8e0 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient(hunting): dump: 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:08.027 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 838221678 0 0) 0x7f5160005490 con 0x7f51800af8e0 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 >> v1:172.21.15.136:6789/0 conn(0x7f518013ef30 legacy=0x7f51801de820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 >> v1:172.21.15.117:6789/0 conn(0x7f51800b0860 legacy=0x7f51801347a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 monclient: found mon.c 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:08.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51801d7350 con 0x7f51800af8e0 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient: _finish_auth 0 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f516effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f51849a3640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.042+0000 7f51849a3640 10 monclient: _renew_subs 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f51849a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51800b2e50 con 0x7f51800af8e0 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f51849a3640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f51849a3640 10 monclient: _renew_subs 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f51849a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5178002080 con 0x7f51800af8e0 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51800b13b0 con 0x7f51800af8e0 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f51849a3640 1 librados: init done 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f516effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:08.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5178001b90 con 0x7f51800af8e0 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f516effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f516effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.043+0000 7f516effd640 10 monclient: dump: 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:08.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.044+0000 7f517cd99640 4 set_mon_vals no callback set 2024-03-29T10:42:08.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.044+0000 7f51849a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.045+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f51780198e0 con 0x7f51800af8e0 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.045+0000 7f51849a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.045+0000 7f51849a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.045+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5164005080 con 0x7f51800af8e0 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.045+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f51780837e0 con 0x7f51800af8e0 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.045+0000 7f516effd640 10 client.4550.objecter ms_dispatch 0x7f5180000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.045+0000 7f516effd640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.045+0000 7f516effd640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.045+0000 7f516effd640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T10:42:08.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.046+0000 7f517dd9b640 10 monclient: get_auth_request con 0x7f516006e0f0 auth_method 0 2024-03-29T10:42:08.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.049+0000 7f516effd640 10 client.4550.objecter ms_handle_connect 0x7f516006e0f0 2024-03-29T10:42:08.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.052+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f517804f120 con 0x7f51800af8e0 2024-03-29T10:42:08.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.052+0000 7f516effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:08.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.052+0000 7f516effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:08.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.389+0000 7f51849a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:08.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.389+0000 7f51849a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:08.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.389+0000 7f51849a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:08.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.389+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5164005330 con 0x7f51800af8e0 2024-03-29T10:42:08.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.390+0000 7f516effd640 1 -- 172.21.15.117:0/2626230897 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f517804f120 con 0x7f51800af8e0 2024-03-29T10:42:08.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.390+0000 7f516effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:08.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.390+0000 7f516effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:08.374 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:42:08.374 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:08.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.393+0000 7f51849a3640 10 librados: watch_flush enter 2024-03-29T10:42:08.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 10 librados: watch_flush exit 2024-03-29T10:42:08.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T10:42:08.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 10 client.4550.objecter successfully canceled tick 2024-03-29T10:42:08.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 >> v1:172.21.15.117:6816/376768171 conn(0x7f516006e0f0 legacy=0x7f51600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:08.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 10 monclient: shutdown 2024-03-29T10:42:08.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:08.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 >> v1:172.21.15.117:6790/0 conn(0x7f51800af8e0 legacy=0x7f51801d6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:08.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 shutdown_connections 2024-03-29T10:42:08.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 >> 172.21.15.117:0/2626230897 conn(0x7f518006b660 msgr2=0x7f51800ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:08.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 shutdown_connections 2024-03-29T10:42:08.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 1 -- 172.21.15.117:0/2626230897 wait complete. 2024-03-29T10:42:08.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:08.394+0000 7f51849a3640 1 librados: shutdown 2024-03-29T10:42:08.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:15.202 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:15.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.369+0000 7f34d324a640 10 monclient: get_monmap_and_config 2024-03-29T10:42:15.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.369+0000 7f34d324a640 10 monclient: build_initial_monmap 2024-03-29T10:42:15.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.369+0000 7f34d324a640 10 monclient: monmap: 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:42:15.370047+0000 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:42:15.370047+0000 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:42:15.353 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:15.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.370+0000 7f34d324a640 1 Processor -- start 2024-03-29T10:42:15.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.370+0000 7f34d324a640 1 -- start start 2024-03-29T10:42:15.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.370+0000 7f34d324a640 10 monclient: init 2024-03-29T10:42:15.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:15.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 10 monclient(hunting): picked mon.noname-c con 0x7f34cc06d9d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 10 monclient(hunting): picked mon.noname-b con 0x7f34cc13ef30 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 10 monclient(hunting): picked mon.noname-a con 0x7f34cc0b07d0 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc0aed10 con 0x7f34cc0b07d0 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc0b0c40 con 0x7f34cc06d9d0 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc0ab1b0 con 0x7f34cc13ef30 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34d324a640 10 monclient(hunting): authenticate will time out at 1256.633655s 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34cbfff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f34cc13ef30 0x7f34cc0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:50038/0 (socket says 172.21.15.117:50038) 2024-03-29T10:42:15.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.371+0000 7f34cbfff640 1 -- 172.21.15.117:0/1493450331 learned_addr learned my addr 172.21.15.117:0/1493450331 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:15.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.373+0000 7f34cb7fe640 1 -- 172.21.15.117:0/1493450331 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755832761 0 0) 0x7f34cc0ab1b0 con 0x7f34cc13ef30 2024-03-29T10:42:15.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.373+0000 7f34cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:15.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.373+0000 7f34cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:15.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.373+0000 7f34cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:15.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.373+0000 7f34cb7fe640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:42:15.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.373+0000 7f34cb7fe640 1 -- 172.21.15.117:0/1493450331 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b4003520 con 0x7f34cc13ef30 2024-03-29T10:42:15.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.373+0000 7f34cb7fe640 1 -- 172.21.15.117:0/1493450331 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3561739523 0 0) 0x7f34b4003520 con 0x7f34cc13ef30 2024-03-29T10:42:15.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.374+0000 7f34cb7fe640 1 -- 172.21.15.117:0/1493450331 >> v1:172.21.15.136:6789/0 conn(0x7f34cc06d9d0 legacy=0x7f34cc13e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:15.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.376+0000 7f34cb7fe640 1 -- 172.21.15.117:0/1493450331 >> v1:172.21.15.117:6789/0 conn(0x7f34cc0b07d0 legacy=0x7f34cc14bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:15.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.376+0000 7f34cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:15.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.376+0000 7f34cb7fe640 1 monclient: found mon.noname-b 2024-03-29T10:42:15.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.376+0000 7f34cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:15.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.376+0000 7f34cb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:42:15.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.376+0000 7f34cb7fe640 1 -- 172.21.15.117:0/1493450331 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc0ab490 con 0x7f34cc13ef30 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 1 -- 172.21.15.117:0/1493450331 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f34c0001690 con 0x7f34cc13ef30 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 10 monclient: dump: 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:15.361 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34d324a640 5 monclient: authenticate success, global_id 4553 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34d324a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 1 -- 172.21.15.117:0/1493450331 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34c0001920 con 0x7f34cc13ef30 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34d324a640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34d324a640 4 set_mon_vals no callback set 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 1 -- 172.21.15.117:0/1493450331 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f34c0001cb0 con 0x7f34cc13ef30 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.377+0000 7f34cb7fe640 10 monclient: dump: 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:15.362 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:15.363 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:15.363 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:15.363 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:15.363 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:15.363 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:15.363 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:15.363 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:15.363 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:15.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.378+0000 7f34d324a640 10 monclient: shutdown 2024-03-29T10:42:15.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.378+0000 7f34d324a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:15.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.378+0000 7f34d324a640 1 -- 172.21.15.117:0/1493450331 >> v1:172.21.15.117:6790/0 conn(0x7f34cc13ef30 legacy=0x7f34cc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:15.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.378+0000 7f34d324a640 1 -- 172.21.15.117:0/1493450331 shutdown_connections 2024-03-29T10:42:15.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.378+0000 7f34d324a640 1 -- 172.21.15.117:0/1493450331 >> 172.21.15.117:0/1493450331 conn(0x7f34cc06b660 msgr2=0x7f34cc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:15.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.379+0000 7f34d324a640 1 -- 172.21.15.117:0/1493450331 shutdown_connections 2024-03-29T10:42:15.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.379+0000 7f34d324a640 1 -- 172.21.15.117:0/1493450331 wait complete. 2024-03-29T10:42:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.380+0000 7f34d324a640 10 monclient: build_initial_monmap 2024-03-29T10:42:15.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.380+0000 7f34d324a640 10 monclient: monmap: 2024-03-29T10:42:15.366 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:15.366 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:15.366 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:42:15.366 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:42:15.366 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:15.366 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.381+0000 7f34d324a640 1 librados: starting msgr at 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.381+0000 7f34d324a640 1 librados: starting objecter 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.381+0000 7f34d324a640 1 Processor -- start 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.381+0000 7f34d324a640 1 -- start start 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.381+0000 7f34d324a640 1 librados: setting wanted keys 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.381+0000 7f34d324a640 1 librados: calling monclient init 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.381+0000 7f34d324a640 10 monclient: init 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 10 monclient(hunting): picked mon.noname-c con 0x7f34cc06d9d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:15.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 10 monclient(hunting): picked mon.noname-b con 0x7f34cc0b07d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 10 monclient(hunting): picked mon.noname-a con 0x7f34cc13ef30 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc06fe70 con 0x7f34cc13ef30 2024-03-29T10:42:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc06e3b0 con 0x7f34cc0b07d0 2024-03-29T10:42:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc070120 con 0x7f34cc06d9d0 2024-03-29T10:42:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:15.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.382+0000 7f34d324a640 10 monclient(hunting): authenticate will time out at 1256.644765s 2024-03-29T10:42:15.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.385+0000 7f34d0fe6640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f34cc06d9d0 0x7f34cc136120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:53862/0 (socket says 172.21.15.117:53862) 2024-03-29T10:42:15.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.385+0000 7f34d0fe6640 1 -- 172.21.15.117:0/2155863317 learned_addr learned my addr 172.21.15.117:0/2155863317 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:15.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.385+0000 7f34c97fa640 10 client.?.objecter ms_handle_connect 0x7f34cc06d9d0 2024-03-29T10:42:15.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.385+0000 7f34c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:15.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123713281 0 0) 0x7f34cc070120 con 0x7f34cc06d9d0 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b8004eb0 con 0x7f34cc06d9d0 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194439020 0 0) 0x7f34b8004eb0 con 0x7f34cc06d9d0 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b8005300 con 0x7f34cc06d9d0 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f34bc001790 con 0x7f34cc06d9d0 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 10 monclient(hunting): dump: 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.386+0000 7f34c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3411131557 0 0) 0x7f34b8005300 con 0x7f34cc06d9d0 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 >> v1:172.21.15.136:6789/0 conn(0x7f34cc0b07d0 legacy=0x7f34cc070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 >> v1:172.21.15.117:6789/0 conn(0x7f34cc13ef30 legacy=0x7f34cc070f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 1 monclient: found mon.c 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:15.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc070400 con 0x7f34cc06d9d0 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 10 monclient: _finish_auth 0 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34d324a640 5 monclient: authenticate success, global_id 4556 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34d324a640 10 monclient: _renew_subs 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34d324a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc06e0d0 con 0x7f34cc06d9d0 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34d324a640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34d324a640 10 monclient: _renew_subs 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34d324a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc071640 con 0x7f34cc06d9d0 2024-03-29T10:42:15.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.387+0000 7f34d324a640 1 librados: init done 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.388+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34bc001d60 con 0x7f34cc06d9d0 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.388+0000 7f34c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.388+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f34bc001790 con 0x7f34cc06d9d0 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.388+0000 7f34c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.388+0000 7f34c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.388+0000 7f34c97fa640 10 monclient: dump: 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.388+0000 7f34caffd640 4 set_mon_vals no callback set 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.388+0000 7f34d324a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.389+0000 7f34d324a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.389+0000 7f34d324a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.389+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac005080 con 0x7f34cc06d9d0 2024-03-29T10:42:15.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.390+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f34bc01a6d0 con 0x7f34cc06d9d0 2024-03-29T10:42:15.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.391+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f34bc054490 con 0x7f34cc06d9d0 2024-03-29T10:42:15.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.391+0000 7f34c97fa640 10 client.4556.objecter ms_dispatch 0x7f34cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:15.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.391+0000 7f34c97fa640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:15.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.391+0000 7f34c97fa640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:15.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.391+0000 7f34c97fa640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T10:42:15.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.392+0000 7f34cbfff640 10 monclient: get_auth_request con 0x7f34b806df90 auth_method 0 2024-03-29T10:42:15.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.392+0000 7f34c97fa640 10 client.4556.objecter ms_handle_connect 0x7f34b806df90 2024-03-29T10:42:15.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.398+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34bc050730 con 0x7f34cc06d9d0 2024-03-29T10:42:15.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.398+0000 7f34c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:15.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.398+0000 7f34c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:15.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.715+0000 7f34d324a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:15.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.715+0000 7f34d324a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:15.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.715+0000 7f34d324a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:15.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.715+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34ac005330 con 0x7f34cc06d9d0 2024-03-29T10:42:15.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.716+0000 7f34c97fa640 1 -- 172.21.15.117:0/2155863317 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f34bc050730 con 0x7f34cc06d9d0 2024-03-29T10:42:15.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.716+0000 7f34c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:15.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.716+0000 7f34c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:15.700 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:42:15.700 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:15.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.720+0000 7f34d324a640 10 librados: watch_flush enter 2024-03-29T10:42:15.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.720+0000 7f34d324a640 10 librados: watch_flush exit 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.720+0000 7f34d324a640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.720+0000 7f34d324a640 10 client.4556.objecter successfully canceled tick 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.721+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 >> v1:172.21.15.117:6816/376768171 conn(0x7f34b806df90 legacy=0x7f34b8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.721+0000 7f34d324a640 10 monclient: shutdown 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.721+0000 7f34d324a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.721+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 >> v1:172.21.15.117:6790/0 conn(0x7f34cc06d9d0 legacy=0x7f34cc136120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.721+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 shutdown_connections 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.721+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 >> 172.21.15.117:0/2155863317 conn(0x7f34cc06b660 msgr2=0x7f34cc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.721+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 shutdown_connections 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.721+0000 7f34d324a640 1 -- 172.21.15.117:0/2155863317 wait complete. 2024-03-29T10:42:15.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:15.721+0000 7f34d324a640 1 librados: shutdown 2024-03-29T10:42:15.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:22.529 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:22.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.693+0000 7f848a30b640 10 monclient: get_monmap_and_config 2024-03-29T10:42:22.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.693+0000 7f848a30b640 10 monclient: build_initial_monmap 2024-03-29T10:42:22.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.693+0000 7f848a30b640 10 monclient: monmap: 2024-03-29T10:42:22.677 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:22.677 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:22.677 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:42:22.694714+0000 2024-03-29T10:42:22.677 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:42:22.694714+0000 2024-03-29T10:42:22.678 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:22.678 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:22.678 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:22.678 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:42:22.678 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:42:22.678 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:22.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.697+0000 7f848a30b640 1 Processor -- start 2024-03-29T10:42:22.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.697+0000 7f848a30b640 1 -- start start 2024-03-29T10:42:22.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.697+0000 7f848a30b640 10 monclient: init 2024-03-29T10:42:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 10 monclient(hunting): picked mon.noname-c con 0x7f847c0389e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 10 monclient(hunting): picked mon.noname-b con 0x7f847c038dc0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 10 monclient(hunting): picked mon.noname-a con 0x7f847c030660 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c0c54c0 con 0x7f847c030660 2024-03-29T10:42:22.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c036430 con 0x7f847c0389e0 2024-03-29T10:42:22.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c039840 con 0x7f847c038dc0 2024-03-29T10:42:22.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:22.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f848a30b640 10 monclient(hunting): authenticate will time out at 1263.960814s 2024-03-29T10:42:22.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.698+0000 7f8488b08640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f847c038dc0 0x7f847c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:42988/0 (socket says 172.21.15.117:42988) 2024-03-29T10:42:22.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.699+0000 7f8488b08640 1 -- 172.21.15.117:0/3480567958 learned_addr learned my addr 172.21.15.117:0/3480567958 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378924700 0 0) 0x7f847c039840 con 0x7f847c038dc0 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8470003520 con 0x7f847c038dc0 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746104384 0 0) 0x7f847c036430 con 0x7f847c0389e0 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:22.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.702+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f847c039840 con 0x7f847c0389e0 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 294381275 0 0) 0x7f847c039840 con 0x7f847c0389e0 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 >> v1:172.21.15.117:6790/0 conn(0x7f847c038dc0 legacy=0x7f847c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 >> v1:172.21.15.117:6789/0 conn(0x7f847c030660 legacy=0x7f847c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 1 monclient: found mon.noname-c 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c035690 con 0x7f847c0389e0 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 10 monclient: _finish_auth 0 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:22.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f84800015f0 con 0x7f847c0389e0 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.703+0000 7f847bfff640 10 monclient: dump: 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:22.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.704+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8480001810 con 0x7f847c0389e0 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.704+0000 7f847bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.704+0000 7f847bfff640 1 -- 172.21.15.117:0/3480567958 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8480001ba0 con 0x7f847c0389e0 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.704+0000 7f847bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.704+0000 7f847bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.704+0000 7f847bfff640 10 monclient: dump: 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:22.692 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:22.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.704+0000 7f848a30b640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:42:22.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.704+0000 7f848a30b640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:22.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.704+0000 7f848a30b640 4 set_mon_vals no callback set 2024-03-29T10:42:22.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.705+0000 7f848a30b640 10 monclient: shutdown 2024-03-29T10:42:22.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.705+0000 7f848a30b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:22.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.705+0000 7f848a30b640 1 -- 172.21.15.117:0/3480567958 >> v1:172.21.15.136:6789/0 conn(0x7f847c0389e0 legacy=0x7f847c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.705+0000 7f848a30b640 1 -- 172.21.15.117:0/3480567958 shutdown_connections 2024-03-29T10:42:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.705+0000 7f848a30b640 1 -- 172.21.15.117:0/3480567958 >> 172.21.15.117:0/3480567958 conn(0x7f847c005c00 msgr2=0x7f847c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:22.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.706+0000 7f848a30b640 1 -- 172.21.15.117:0/3480567958 shutdown_connections 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.706+0000 7f848a30b640 1 -- 172.21.15.117:0/3480567958 wait complete. 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.708+0000 7f848a30b640 10 monclient: build_initial_monmap 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.708+0000 7f848a30b640 10 monclient: monmap: 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:22.695 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.708+0000 7f848a30b640 1 librados: starting msgr at 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.708+0000 7f848a30b640 1 librados: starting objecter 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.708+0000 7f848a30b640 1 Processor -- start 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 1 -- start start 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 1 librados: setting wanted keys 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 1 librados: calling monclient init 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 10 monclient: init 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 10 monclient(hunting): picked mon.noname-a con 0x7f847c030660 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 10 monclient(hunting): picked mon.noname-c con 0x7f847c0389e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 10 monclient(hunting): picked mon.noname-b con 0x7f847c009770 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:22.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.709+0000 7f848a30b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c00a230 con 0x7f847c030660 2024-03-29T10:42:22.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.710+0000 7f848a30b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c02f700 con 0x7f847c009770 2024-03-29T10:42:22.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.710+0000 7f848a30b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c00a480 con 0x7f847c0389e0 2024-03-29T10:42:22.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.710+0000 7f848a30b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:22.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.710+0000 7f848a30b640 10 monclient(hunting): authenticate will time out at 1263.972195s 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.712+0000 7f8488b08640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f847c0389e0 0x7f847c00ae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44770/0 (socket says 172.21.15.117:44770) 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.712+0000 7f8488b08640 1 -- 172.21.15.117:0/2329373860 learned_addr learned my addr 172.21.15.117:0/2329373860 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.712+0000 7f8479ffb640 10 client.?.objecter ms_handle_connect 0x7f847c0389e0 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.712+0000 7f8479ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 client.?.objecter ms_handle_connect 0x7f847c030660 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597385839 0 0) 0x7f847c00a230 con 0x7f847c030660 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8468004eb0 con 0x7f847c030660 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107232626 0 0) 0x7f847c00a480 con 0x7f847c0389e0 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f847c00a230 con 0x7f847c0389e0 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2321917311 0 0) 0x7f8468004eb0 con 0x7f847c030660 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f847c00a480 con 0x7f847c030660 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.713+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8480001530 con 0x7f847c030660 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 10 monclient(hunting): dump: 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1370419892 0 0) 0x7f847c00a480 con 0x7f847c030660 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 >> v1:172.21.15.117:6790/0 conn(0x7f847c0389e0 legacy=0x7f847c00ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 >> v1:172.21.15.136:6789/0 conn(0x7f847c009770 legacy=0x7f847c174040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 1 monclient: found mon.a 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.714+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c00a760 con 0x7f847c030660 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f8479ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f8479ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f8479ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f848a30b640 5 monclient: authenticate success, global_id 4500 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f848a30b640 10 monclient: _renew_subs 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f848a30b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f848a30b640 1 -- 172.21.15.117:0/2329373860 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c00aa40 con 0x7f847c030660 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f848a30b640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f848a30b640 10 monclient: _renew_subs 2024-03-29T10:42:22.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.715+0000 7f848a30b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.716+0000 7f848a30b640 1 -- 172.21.15.117:0/2329373860 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c0c80f0 con 0x7f847c030660 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.716+0000 7f848a30b640 1 librados: init done 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.716+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8480001690 con 0x7f847c030660 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.716+0000 7f8479ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.716+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8480003a80 con 0x7f847c030660 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.716+0000 7f8479ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.716+0000 7f847b7fe640 4 set_mon_vals no callback set 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.716+0000 7f8479ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.716+0000 7f8479ffb640 10 monclient: dump: 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:22.706 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:22.707 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:22.707 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:22.707 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:22.707 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:22.707 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:22.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.718+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f8480003be0 con 0x7f847c030660 2024-03-29T10:42:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.719+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f8480083910 con 0x7f847c030660 2024-03-29T10:42:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.719+0000 7f8479ffb640 10 client.4500.objecter ms_dispatch 0x7f8484000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.719+0000 7f8479ffb640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:22.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.719+0000 7f8479ffb640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.720+0000 7f8479ffb640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.720+0000 7f846f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.720+0000 7f846f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.720+0000 7f846f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.720+0000 7f846f7fe640 1 -- 172.21.15.117:0/2329373860 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c007b90 con 0x7f847c030660 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.720+0000 7f8488b08640 10 monclient: get_auth_request con 0x7f846806dd00 auth_method 0 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.720+0000 7f8479ffb640 10 client.4500.objecter ms_handle_connect 0x7f846806dd00 2024-03-29T10:42:22.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.728+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f848004f330 con 0x7f847c030660 2024-03-29T10:42:22.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.729+0000 7f8479ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:22.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:22.729+0000 7f8479ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:23.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.056+0000 7f846f7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:23.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.056+0000 7f846f7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:23.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.056+0000 7f846f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:23.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.056+0000 7f846f7fe640 1 -- 172.21.15.117:0/2329373860 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f847c02f890 con 0x7f847c030660 2024-03-29T10:42:23.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.060+0000 7f8479ffb640 1 -- 172.21.15.117:0/2329373860 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f847c02f890 con 0x7f847c030660 2024-03-29T10:42:23.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.060+0000 7f8479ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:23.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.060+0000 7f8479ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:23.044 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:42:23.045 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:23.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.064+0000 7f846f7fe640 10 librados: watch_flush enter 2024-03-29T10:42:23.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 10 librados: watch_flush exit 2024-03-29T10:42:23.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T10:42:23.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 10 client.4500.objecter successfully canceled tick 2024-03-29T10:42:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 1 -- 172.21.15.117:0/2329373860 >> v1:172.21.15.117:6816/376768171 conn(0x7f846806dd00 legacy=0x7f84680701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 10 monclient: shutdown 2024-03-29T10:42:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 1 -- 172.21.15.117:0/2329373860 >> v1:172.21.15.117:6789/0 conn(0x7f847c030660 legacy=0x7f847c02fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 1 -- 172.21.15.117:0/2329373860 shutdown_connections 2024-03-29T10:42:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 1 -- 172.21.15.117:0/2329373860 >> 172.21.15.117:0/2329373860 conn(0x7f847c005c00 msgr2=0x7f847c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 1 -- 172.21.15.117:0/2329373860 shutdown_connections 2024-03-29T10:42:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.065+0000 7f846f7fe640 1 -- 172.21.15.117:0/2329373860 wait complete. 2024-03-29T10:42:23.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:23.066+0000 7f846f7fe640 1 librados: shutdown 2024-03-29T10:42:23.066 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:29.872 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:30.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.039+0000 7fdbe0f65640 10 monclient: get_monmap_and_config 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.039+0000 7fdbe0f65640 10 monclient: build_initial_monmap 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.039+0000 7fdbe0f65640 10 monclient: monmap: 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:42:30.040093+0000 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:42:30.040093+0000 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:42:30.023 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:30.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.040+0000 7fdbe0f65640 1 Processor -- start 2024-03-29T10:42:30.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.040+0000 7fdbe0f65640 1 -- start start 2024-03-29T10:42:30.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.040+0000 7fdbe0f65640 10 monclient: init 2024-03-29T10:42:30.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.041+0000 7fdbe0f65640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:30.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.041+0000 7fdbe0f65640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbe0f65640 10 monclient(hunting): picked mon.noname-c con 0x7fdbd40389e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbe0f65640 10 monclient(hunting): picked mon.noname-b con 0x7fdbd4038dc0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbe0f65640 10 monclient(hunting): picked mon.noname-a con 0x7fdbd4030660 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbe0f65640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40c54c0 con 0x7fdbd4030660 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbe0f65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd4036430 con 0x7fdbd40389e0 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbe0f65640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd4039840 con 0x7fdbd4038dc0 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbe0f65640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbe0f65640 10 monclient(hunting): authenticate will time out at 1271.304834s 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbdaffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fdbd4038dc0 0x7fdbd40cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:52800/0 (socket says 172.21.15.117:52800) 2024-03-29T10:42:30.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.042+0000 7fdbdaffd640 1 -- 172.21.15.117:0/1773651374 learned_addr learned my addr 172.21.15.117:0/1773651374 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:30.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.044+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459264399 0 0) 0x7fdbd4039840 con 0x7fdbd4038dc0 2024-03-29T10:42:30.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.044+0000 7fdbda7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:30.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.044+0000 7fdbda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:30.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.044+0000 7fdbda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:30.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.044+0000 7fdbda7fc640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:42:30.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.044+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbcc003520 con 0x7fdbd4038dc0 2024-03-29T10:42:30.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.046+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644225371 0 0) 0x7fdbd40c54c0 con 0x7fdbd4030660 2024-03-29T10:42:30.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.046+0000 7fdbda7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:30.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.046+0000 7fdbda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:30.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.046+0000 7fdbda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.046+0000 7fdbda7fc640 10 monclient(hunting): my global_id is 4503 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.046+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd4039840 con 0x7fdbd4030660 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801513416 0 0) 0x7fdbd4036430 con 0x7fdbd40389e0 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd40c54c0 con 0x7fdbd40389e0 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647858033 0 0) 0x7fdbcc003520 con 0x7fdbd4038dc0 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 >> v1:172.21.15.136:6789/0 conn(0x7fdbd40389e0 legacy=0x7fdbd4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 >> v1:172.21.15.117:6789/0 conn(0x7fdbd4030660 legacy=0x7fdbd40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 1 monclient: found mon.noname-b 2024-03-29T10:42:30.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd4035690 con 0x7fdbd4038dc0 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdbc8001610 con 0x7fdbd4038dc0 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient: dump: 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.035 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbe0f65640 5 monclient: authenticate success, global_id 4565 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbe0f65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbc8001770 con 0x7fdbd4038dc0 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbe0f65640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.047+0000 7fdbe0f65640 4 set_mon_vals no callback set 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.048+0000 7fdbda7fc640 1 -- 172.21.15.117:0/1773651374 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdbc8001bc0 con 0x7fdbd4038dc0 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.048+0000 7fdbda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:30.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.048+0000 7fdbda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.048+0000 7fdbda7fc640 10 monclient: dump: 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.048+0000 7fdbe0f65640 10 monclient: shutdown 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.048+0000 7fdbe0f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:30.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.048+0000 7fdbe0f65640 1 -- 172.21.15.117:0/1773651374 >> v1:172.21.15.117:6790/0 conn(0x7fdbd4038dc0 legacy=0x7fdbd40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.051+0000 7fdbe0f65640 1 -- 172.21.15.117:0/1773651374 shutdown_connections 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.051+0000 7fdbe0f65640 1 -- 172.21.15.117:0/1773651374 >> 172.21.15.117:0/1773651374 conn(0x7fdbd4005c00 msgr2=0x7fdbd4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.051+0000 7fdbe0f65640 1 -- 172.21.15.117:0/1773651374 shutdown_connections 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.051+0000 7fdbe0f65640 1 -- 172.21.15.117:0/1773651374 wait complete. 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.052+0000 7fdbe0f65640 10 monclient: build_initial_monmap 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.052+0000 7fdbe0f65640 10 monclient: monmap: 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:30.040 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.052+0000 7fdbe0f65640 1 librados: starting msgr at 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.052+0000 7fdbe0f65640 1 librados: starting objecter 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.053+0000 7fdbe0f65640 1 Processor -- start 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.053+0000 7fdbe0f65640 1 -- start start 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.053+0000 7fdbe0f65640 1 librados: setting wanted keys 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.053+0000 7fdbe0f65640 1 librados: calling monclient init 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.053+0000 7fdbe0f65640 10 monclient: init 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.054+0000 7fdbe0f65640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.054+0000 7fdbe0f65640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:30.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.054+0000 7fdbe0f65640 10 monclient(hunting): picked mon.noname-c con 0x7fdbd4030660 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.054+0000 7fdbdb7fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fdbd4030660 0x7fdbd415cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:59306/0 (socket says 172.21.15.117:59306) 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.054+0000 7fdbdb7fe640 1 -- 172.21.15.117:0/862693013 learned_addr learned my addr 172.21.15.117:0/862693013 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.055+0000 7fdbc7fff640 10 client.?.objecter ms_handle_connect 0x7fdbd4030660 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.055+0000 7fdbc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.056+0000 7fdbe0f65640 10 monclient(hunting): picked mon.noname-a con 0x7fdbd40389e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.056+0000 7fdbe0f65640 10 monclient(hunting): picked mon.noname-b con 0x7fdbd415d4d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.056+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd402ea00 con 0x7fdbd40389e0 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.056+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd402ece0 con 0x7fdbd415d4d0 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.056+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd415c8a0 con 0x7fdbd4030660 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.056+0000 7fdbe0f65640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:30.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.056+0000 7fdbe0f65640 10 monclient(hunting): authenticate will time out at 1271.318492s 2024-03-29T10:42:30.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 client.?.objecter ms_handle_connect 0x7fdbd40389e0 2024-03-29T10:42:30.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:30.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971392104 0 0) 0x7fdbd402ea00 con 0x7fdbd40389e0 2024-03-29T10:42:30.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:30.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc0004eb0 con 0x7fdbd40389e0 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 client.?.objecter ms_handle_connect 0x7fdbd415d4d0 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431016626 0 0) 0x7fdbd415c8a0 con 0x7fdbd4030660 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:42:30.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd402ea00 con 0x7fdbd4030660 2024-03-29T10:42:30.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666298063 0 0) 0x7fdbd402ece0 con 0x7fdbd415d4d0 2024-03-29T10:42:30.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:30.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:30.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:30.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:42:30.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd415c8a0 con 0x7fdbd415d4d0 2024-03-29T10:42:30.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.061+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733193828 0 0) 0x7fdbd402ea00 con 0x7fdbd4030660 2024-03-29T10:42:30.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbd402ece0 con 0x7fdbd4030660 2024-03-29T10:42:30.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3550056318 0 0) 0x7fdbc0004eb0 con 0x7fdbd40389e0 2024-03-29T10:42:30.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbd402ea00 con 0x7fdbd40389e0 2024-03-29T10:42:30.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdbd00018b0 con 0x7fdbd4030660 2024-03-29T10:42:30.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:30.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:30.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient(hunting): dump: 2024-03-29T10:42:30.051 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:30.051 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:30.051 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.051 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdbc8001a80 con 0x7fdbd40389e0 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient(hunting): dump: 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.052 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 817210109 0 0) 0x7fdbd402ece0 con 0x7fdbd4030660 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 >> v1:172.21.15.136:6789/0 conn(0x7fdbd415d4d0 legacy=0x7fdbd41681c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 >> v1:172.21.15.117:6789/0 conn(0x7fdbd40389e0 legacy=0x7fdbd402f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 monclient: found mon.c 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:30.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd402f100 con 0x7fdbd4030660 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient: _finish_auth 0 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.062+0000 7fdbc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd0003610 con 0x7fdbd4030660 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbc7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fdbd00039a0 con 0x7fdbd4030660 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbc7fff640 10 monclient: dump: 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:30.054 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbe0f65640 5 monclient: authenticate success, global_id 4568 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbe0f65640 10 monclient: _renew_subs 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbe0f65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd402fe70 con 0x7fdbd4030660 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbd9ffb640 4 set_mon_vals no callback set 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbe0f65640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:30.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbe0f65640 10 monclient: _renew_subs 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbe0f65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd40c8250 con 0x7fdbd4030660 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.063+0000 7fdbe0f65640 1 librados: init done 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.064+0000 7fdbe0f65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.064+0000 7fdbe0f65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.064+0000 7fdbe0f65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.064+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd4007b70 con 0x7fdbd4030660 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.065+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fdbd0003b00 con 0x7fdbd4030660 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.066+0000 7fdbdaffd640 10 monclient: get_auth_request con 0x7fdbc006df20 auth_method 0 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.066+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fdbd0084870 con 0x7fdbd4030660 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.066+0000 7fdbc7fff640 10 client.4568.objecter ms_dispatch 0x7fdbdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.066+0000 7fdbc7fff640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.066+0000 7fdbc7fff640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.066+0000 7fdbc7fff640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T10:42:30.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.067+0000 7fdbc7fff640 10 client.4568.objecter ms_handle_connect 0x7fdbc006df20 2024-03-29T10:42:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.073+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbd0050150 con 0x7fdbd4030660 2024-03-29T10:42:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.073+0000 7fdbc7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:30.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.073+0000 7fdbc7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:30.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.410+0000 7fdbe0f65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:30.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.410+0000 7fdbe0f65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:30.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.410+0000 7fdbe0f65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:30.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.410+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbd40301c0 con 0x7fdbd4030660 2024-03-29T10:42:30.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.411+0000 7fdbc7fff640 1 -- 172.21.15.117:0/862693013 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fdbd0052080 con 0x7fdbd4030660 2024-03-29T10:42:30.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.411+0000 7fdbc7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:30.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.411+0000 7fdbc7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:30.395 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:42:30.395 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:30.397 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.414+0000 7fdbe0f65640 10 librados: watch_flush enter 2024-03-29T10:42:30.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.415+0000 7fdbe0f65640 10 librados: watch_flush exit 2024-03-29T10:42:30.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.415+0000 7fdbe0f65640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T10:42:30.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.415+0000 7fdbe0f65640 10 client.4568.objecter successfully canceled tick 2024-03-29T10:42:30.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.416+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 >> v1:172.21.15.117:6816/376768171 conn(0x7fdbc006df20 legacy=0x7fdbc00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:30.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.416+0000 7fdbe0f65640 10 monclient: shutdown 2024-03-29T10:42:30.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.416+0000 7fdbe0f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:30.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.416+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 >> v1:172.21.15.117:6790/0 conn(0x7fdbd4030660 legacy=0x7fdbd415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:30.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.416+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 shutdown_connections 2024-03-29T10:42:30.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.416+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 >> 172.21.15.117:0/862693013 conn(0x7fdbd4005c00 msgr2=0x7fdbd4034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:30.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.417+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 shutdown_connections 2024-03-29T10:42:30.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.417+0000 7fdbe0f65640 1 -- 172.21.15.117:0/862693013 wait complete. 2024-03-29T10:42:30.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:30.418+0000 7fdbe0f65640 1 librados: shutdown 2024-03-29T10:42:30.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:37.227 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:37.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.398+0000 7f502f17a640 10 monclient: get_monmap_and_config 2024-03-29T10:42:37.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.398+0000 7f502f17a640 10 monclient: build_initial_monmap 2024-03-29T10:42:37.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.398+0000 7f502f17a640 10 monclient: monmap: 2024-03-29T10:42:37.383 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:37.383 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:37.383 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:42:37.400326+0000 2024-03-29T10:42:37.383 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:42:37.400326+0000 2024-03-29T10:42:37.383 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:37.384 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:37.384 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:37.384 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:42:37.384 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:42:37.384 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:37.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.399+0000 7f502f17a640 1 Processor -- start 2024-03-29T10:42:37.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.399+0000 7f502f17a640 1 -- start start 2024-03-29T10:42:37.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.399+0000 7f502f17a640 10 monclient: init 2024-03-29T10:42:37.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.400+0000 7f502f17a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:37.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.400+0000 7f502f17a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:37.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.404+0000 7f502f17a640 10 monclient(hunting): picked mon.noname-a con 0x7f50280b0770 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:37.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.404+0000 7f502cf16640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f50280b0770 0x7f5028149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:53697/0 (socket says 172.21.15.117:53697) 2024-03-29T10:42:37.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f502f17a640 10 monclient(hunting): picked mon.noname-c con 0x7f502814a700 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:37.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f502f17a640 10 monclient(hunting): picked mon.noname-b con 0x7f50280ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:37.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f502f17a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50280aba70 con 0x7f50280b0770 2024-03-29T10:42:37.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f502f17a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50280aecb0 con 0x7f502814a700 2024-03-29T10:42:37.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f502f17a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50280af150 con 0x7f50280ac5a0 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f502f17a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f502cf16640 1 -- 172.21.15.117:0/1562792806 learned_addr learned my addr 172.21.15.117:0/1562792806 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f502f17a640 10 monclient(hunting): authenticate will time out at 1278.669219s 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f50277fe640 1 -- 172.21.15.117:0/1562792806 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486621838 0 0) 0x7f50280aba70 con 0x7f50280b0770 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f50277fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f50277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f50277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f50277fe640 10 monclient(hunting): my global_id is 4509 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f50277fe640 1 -- 172.21.15.117:0/1562792806 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5010003520 con 0x7f50280b0770 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f50277fe640 1 -- 172.21.15.117:0/1562792806 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2440097120 0 0) 0x7f5010003520 con 0x7f50280b0770 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f50277fe640 1 -- 172.21.15.117:0/1562792806 >> v1:172.21.15.117:6790/0 conn(0x7f50280ac5a0 legacy=0x7f5028151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.406+0000 7f50277fe640 1 -- 172.21.15.117:0/1562792806 >> v1:172.21.15.136:6789/0 conn(0x7f502814a700 legacy=0x7f50280abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:37.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:37.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 1 monclient: found mon.noname-a 2024-03-29T10:42:37.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:37.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:42:37.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 1 -- 172.21.15.117:0/1562792806 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50280ac9d0 con 0x7f50280b0770 2024-03-29T10:42:37.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:37.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:37.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 1 -- 172.21.15.117:0/1562792806 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5020001a10 con 0x7f50280b0770 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 10 monclient: dump: 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:37.399 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.407+0000 7f50277fe640 1 -- 172.21.15.117:0/1562792806 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5020001ca0 con 0x7f50280b0770 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.408+0000 7f50277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.408+0000 7f50277fe640 1 -- 172.21.15.117:0/1562792806 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f5020002070 con 0x7f50280b0770 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.408+0000 7f50277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.408+0000 7f50277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.408+0000 7f50277fe640 10 monclient: dump: 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:37.400 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.408+0000 7f502f17a640 5 monclient: authenticate success, global_id 4509 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.408+0000 7f502f17a640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.408+0000 7f502f17a640 4 set_mon_vals no callback set 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.409+0000 7f502f17a640 10 monclient: shutdown 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.409+0000 7f502f17a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:37.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.409+0000 7f502f17a640 1 -- 172.21.15.117:0/1562792806 >> v1:172.21.15.117:6789/0 conn(0x7f50280b0770 legacy=0x7f5028149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.409+0000 7f502f17a640 1 -- 172.21.15.117:0/1562792806 shutdown_connections 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.409+0000 7f502f17a640 1 -- 172.21.15.117:0/1562792806 >> 172.21.15.117:0/1562792806 conn(0x7f502806b660 msgr2=0x7f502806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.411+0000 7f502f17a640 1 -- 172.21.15.117:0/1562792806 shutdown_connections 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.411+0000 7f502f17a640 1 -- 172.21.15.117:0/1562792806 wait complete. 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.412+0000 7f502f17a640 10 monclient: build_initial_monmap 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.412+0000 7f502f17a640 10 monclient: monmap: 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:37.402 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.413+0000 7f502f17a640 1 librados: starting msgr at 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.413+0000 7f502f17a640 1 librados: starting objecter 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.413+0000 7f502f17a640 1 Processor -- start 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.413+0000 7f502f17a640 1 -- start start 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.413+0000 7f502f17a640 1 librados: setting wanted keys 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.413+0000 7f502f17a640 1 librados: calling monclient init 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.413+0000 7f502f17a640 10 monclient: init 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.414+0000 7f502f17a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:37.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.414+0000 7f502f17a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:37.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.414+0000 7f502f17a640 10 monclient(hunting): picked mon.noname-a con 0x7f50280ac5a0 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:37.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f502f17a640 10 monclient(hunting): picked mon.noname-b con 0x7f50280b0770 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:37.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f5027fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f50280b0770 0x7f50281db6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:47217/0 (socket says 172.21.15.117:47217) 2024-03-29T10:42:37.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f502f17a640 10 monclient(hunting): picked mon.noname-c con 0x7f502814a700 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:37.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f502f17a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50281239d0 con 0x7f50280ac5a0 2024-03-29T10:42:37.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f502f17a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50281265c0 con 0x7f50280b0770 2024-03-29T10:42:37.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f502f17a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5028145a50 con 0x7f502814a700 2024-03-29T10:42:37.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f502f17a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:37.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f502f17a640 10 monclient(hunting): authenticate will time out at 1278.680515s 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f5027fff640 1 -- 172.21.15.117:0/970056073 learned_addr learned my addr 172.21.15.117:0/970056073 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f50257fa640 10 client.?.objecter ms_handle_connect 0x7f50280b0770 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.417+0000 7f50257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200403249 0 0) 0x7f50281265c0 con 0x7f50280b0770 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5014004eb0 con 0x7f50280b0770 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 client.?.objecter ms_handle_connect 0x7f502814a700 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:37.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 client.?.objecter ms_handle_connect 0x7f50280ac5a0 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178377527 0 0) 0x7f50281239d0 con 0x7f50280ac5a0 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50140051d0 con 0x7f50280ac5a0 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 526956078 0 0) 0x7f5014004eb0 con 0x7f50280b0770 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50281239d0 con 0x7f50280b0770 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f501c001790 con 0x7f50280b0770 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:37.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): dump: 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 381346913 0 0) 0x7f50281239d0 con 0x7f50280b0770 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.418+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 >> v1:172.21.15.117:6790/0 conn(0x7f502814a700 legacy=0x7f50281e5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:37.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.419+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 >> v1:172.21.15.117:6789/0 conn(0x7f50280ac5a0 legacy=0x7f5028125eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.419+0000 7f50257fa640 10 monclient: _finish_hunting 0 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.419+0000 7f50257fa640 1 monclient: found mon.b 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.419+0000 7f50257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.419+0000 7f50257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.419+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5028145be0 con 0x7f50280b0770 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.419+0000 7f50257fa640 10 monclient: _finish_auth 0 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.419+0000 7f50257fa640 10 monclient: _check_auth_tickets 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.419+0000 7f50257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.420+0000 7f502f17a640 5 monclient: authenticate success, global_id 4483 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.420+0000 7f502f17a640 10 monclient: _renew_subs 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.420+0000 7f502f17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.420+0000 7f502f17a640 1 -- 172.21.15.117:0/970056073 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5028126c80 con 0x7f50280b0770 2024-03-29T10:42:37.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f502f17a640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f502f17a640 10 monclient: _renew_subs 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f502f17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f502f17a640 1 -- 172.21.15.117:0/970056073 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50281dbfd0 con 0x7f50280b0770 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f502f17a640 1 librados: init done 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f501c001ca0 con 0x7f50280b0770 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f50257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f501c001790 con 0x7f50280b0770 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f50257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f50257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f50257fa640 10 monclient: dump: 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:37.414 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:37.415 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:37.415 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:37.415 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:37.415 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:37.415 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:37.415 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:37.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.421+0000 7f5026ffd640 4 set_mon_vals no callback set 2024-03-29T10:42:37.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.422+0000 7f500effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:37.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f501c01a750 con 0x7f50280b0770 2024-03-29T10:42:37.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f500effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:37.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f500effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:37.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f500effd640 1 -- 172.21.15.117:0/970056073 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5028075f10 con 0x7f50280b0770 2024-03-29T10:42:37.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f502cf16640 10 monclient: get_auth_request con 0x7f501406e2f0 auth_method 0 2024-03-29T10:42:37.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f501c0015b0 con 0x7f50280b0770 2024-03-29T10:42:37.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f50257fa640 10 client.4483.objecter ms_dispatch 0x7f5028000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:37.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f50257fa640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:37.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f50257fa640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:37.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.424+0000 7f50257fa640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T10:42:37.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.426+0000 7f50257fa640 10 client.4483.objecter ms_handle_connect 0x7f501406e2f0 2024-03-29T10:42:37.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.434+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f501c050570 con 0x7f50280b0770 2024-03-29T10:42:37.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.435+0000 7f50257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:37.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.435+0000 7f50257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:37.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.750+0000 7f500effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:37.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.750+0000 7f500effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:37.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.750+0000 7f500effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:37.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.750+0000 7f500effd640 1 -- 172.21.15.117:0/970056073 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f502806ab70 con 0x7f50280b0770 2024-03-29T10:42:37.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.751+0000 7f50257fa640 1 -- 172.21.15.117:0/970056073 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f501c050570 con 0x7f50280b0770 2024-03-29T10:42:37.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.751+0000 7f50257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:37.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.751+0000 7f50257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:37.736 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:42:37.736 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:37.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.756+0000 7f500effd640 10 librados: watch_flush enter 2024-03-29T10:42:37.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.756+0000 7f500effd640 10 librados: watch_flush exit 2024-03-29T10:42:37.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.756+0000 7f500effd640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T10:42:37.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.756+0000 7f500effd640 10 client.4483.objecter successfully canceled tick 2024-03-29T10:42:37.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.757+0000 7f500effd640 1 -- 172.21.15.117:0/970056073 >> v1:172.21.15.117:6816/376768171 conn(0x7f501406e2f0 legacy=0x7f50140707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:37.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.758+0000 7f500effd640 10 monclient: shutdown 2024-03-29T10:42:37.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.758+0000 7f500effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:37.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.758+0000 7f500effd640 1 -- 172.21.15.117:0/970056073 >> v1:172.21.15.136:6789/0 conn(0x7f50280b0770 legacy=0x7f50281db6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:37.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.758+0000 7f500effd640 1 -- 172.21.15.117:0/970056073 shutdown_connections 2024-03-29T10:42:37.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.758+0000 7f500effd640 1 -- 172.21.15.117:0/970056073 >> 172.21.15.117:0/970056073 conn(0x7f502806b660 msgr2=0x7f50280b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:37.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.760+0000 7f500effd640 1 -- 172.21.15.117:0/970056073 shutdown_connections 2024-03-29T10:42:37.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.760+0000 7f500effd640 1 -- 172.21.15.117:0/970056073 wait complete. 2024-03-29T10:42:37.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:37.761+0000 7f500effd640 1 librados: shutdown 2024-03-29T10:42:37.763 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:44.570 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:44.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.729+0000 7f6e9eb90640 10 monclient: get_monmap_and_config 2024-03-29T10:42:44.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.729+0000 7f6e9eb90640 10 monclient: build_initial_monmap 2024-03-29T10:42:44.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.729+0000 7f6e9eb90640 10 monclient: monmap: 2024-03-29T10:42:44.712 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:44.713 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:44.713 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:42:44.730162+0000 2024-03-29T10:42:44.713 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:42:44.730162+0000 2024-03-29T10:42:44.713 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:44.713 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:44.713 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:44.713 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:42:44.713 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:42:44.713 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:44.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.730+0000 7f6e9eb90640 1 Processor -- start 2024-03-29T10:42:44.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.730+0000 7f6e9eb90640 1 -- start start 2024-03-29T10:42:44.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.730+0000 7f6e9eb90640 10 monclient: init 2024-03-29T10:42:44.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.731+0000 7f6e9eb90640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:44.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.731+0000 7f6e9eb90640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:44.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.731+0000 7f6e9eb90640 10 monclient(hunting): picked mon.noname-b con 0x7f6e981422b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:44.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.731+0000 7f6e9eb90640 10 monclient(hunting): picked mon.noname-a con 0x7f6e98142690 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:44.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.732+0000 7f6e9eb90640 10 monclient(hunting): picked mon.noname-c con 0x7f6e980b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:44.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.732+0000 7f6e9eb90640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e98142c80 con 0x7f6e98142690 2024-03-29T10:42:44.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.732+0000 7f6e9eb90640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980aae40 con 0x7f6e980b17d0 2024-03-29T10:42:44.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.732+0000 7f6e9eb90640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9813fbd0 con 0x7f6e981422b0 2024-03-29T10:42:44.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.732+0000 7f6e9eb90640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:44.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.732+0000 7f6e9eb90640 10 monclient(hunting): authenticate will time out at 1285.994279s 2024-03-29T10:42:44.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.733+0000 7f6e9c92c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f6e981422b0 0x7f6e9813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:53922/0 (socket says 172.21.15.117:53922) 2024-03-29T10:42:44.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.733+0000 7f6e9c92c640 1 -- 172.21.15.117:0/2140081714 learned_addr learned my addr 172.21.15.117:0/2140081714 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:44.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266429088 0 0) 0x7f6e9813fbd0 con 0x7f6e981422b0 2024-03-29T10:42:44.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:44.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:44.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:44.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:42:44.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e88003520 con 0x7f6e981422b0 2024-03-29T10:42:44.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231935035 0 0) 0x7f6e98142c80 con 0x7f6e98142690 2024-03-29T10:42:44.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:44.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:44.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:44.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 10 monclient(hunting): my global_id is 4515 2024-03-29T10:42:44.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e9813fbd0 con 0x7f6e98142690 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 804911334 0 0) 0x7f6e88003520 con 0x7f6e981422b0 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 >> v1:172.21.15.136:6789/0 conn(0x7f6e980b17d0 legacy=0x7f6e980b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.736+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 >> v1:172.21.15.117:6789/0 conn(0x7f6e98142690 legacy=0x7f6e980b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 1 monclient: found mon.noname-b 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98151940 con 0x7f6e981422b0 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e8c001690 con 0x7f6e981422b0 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:44.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: dump: 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e8c001870 con 0x7f6e981422b0 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:44.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 1 -- 172.21.15.117:0/2140081714 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e8c001c00 con 0x7f6e981422b0 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e977fe640 10 monclient: dump: 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:44.723 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e9eb90640 5 monclient: authenticate success, global_id 4574 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e9eb90640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.737+0000 7f6e9eb90640 4 set_mon_vals no callback set 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.738+0000 7f6e9eb90640 10 monclient: shutdown 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.738+0000 7f6e9eb90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.738+0000 7f6e9eb90640 1 -- 172.21.15.117:0/2140081714 >> v1:172.21.15.117:6790/0 conn(0x7f6e981422b0 legacy=0x7f6e9813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.739+0000 7f6e9eb90640 1 -- 172.21.15.117:0/2140081714 shutdown_connections 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.739+0000 7f6e9eb90640 1 -- 172.21.15.117:0/2140081714 >> 172.21.15.117:0/2140081714 conn(0x7f6e98066a00 msgr2=0x7f6e98066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.739+0000 7f6e9eb90640 1 -- 172.21.15.117:0/2140081714 shutdown_connections 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.739+0000 7f6e9eb90640 1 -- 172.21.15.117:0/2140081714 wait complete. 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.740+0000 7f6e9eb90640 10 monclient: build_initial_monmap 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.740+0000 7f6e9eb90640 10 monclient: monmap: 2024-03-29T10:42:44.725 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.741+0000 7f6e9eb90640 1 librados: starting msgr at 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.741+0000 7f6e9eb90640 1 librados: starting objecter 2024-03-29T10:42:44.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.741+0000 7f6e9eb90640 1 Processor -- start 2024-03-29T10:42:44.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.743+0000 7f6e9eb90640 1 -- start start 2024-03-29T10:42:44.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.743+0000 7f6e9eb90640 1 librados: setting wanted keys 2024-03-29T10:42:44.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.743+0000 7f6e9eb90640 1 librados: calling monclient init 2024-03-29T10:42:44.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.743+0000 7f6e9eb90640 10 monclient: init 2024-03-29T10:42:44.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:44.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:44.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 10 monclient(hunting): picked mon.noname-b con 0x7f6e980b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:44.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 10 monclient(hunting): picked mon.noname-c con 0x7f6e981422b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:44.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 10 monclient(hunting): picked mon.noname-a con 0x7f6e98142690 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:44.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9806e9b0 con 0x7f6e98142690 2024-03-29T10:42:44.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9806ec90 con 0x7f6e980b17d0 2024-03-29T10:42:44.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9806ef70 con 0x7f6e981422b0 2024-03-29T10:42:44.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:44.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.744+0000 7f6e9eb90640 10 monclient(hunting): authenticate will time out at 1286.006573s 2024-03-29T10:42:44.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.748+0000 7f6e97fff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f6e981422b0 0x7f6e9814f270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:55568/0 (socket says 172.21.15.117:55568) 2024-03-29T10:42:44.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.748+0000 7f6e97fff640 1 -- 172.21.15.117:0/312112508 learned_addr learned my addr 172.21.15.117:0/312112508 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:44.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.748+0000 7f6e957fa640 10 client.?.objecter ms_handle_connect 0x7f6e981422b0 2024-03-29T10:42:44.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.748+0000 7f6e957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:44.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244897145 0 0) 0x7f6e9806ef70 con 0x7f6e981422b0 2024-03-29T10:42:44.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:44.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:44.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:44.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:42:44.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e7c004eb0 con 0x7f6e981422b0 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3820090809 0 0) 0x7f6e7c004eb0 con 0x7f6e981422b0 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e9806ef70 con 0x7f6e981422b0 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e84001990 con 0x7f6e981422b0 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 10 monclient(hunting): dump: 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:44.733 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:44.734 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:44.734 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:44.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:44.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3516273596 0 0) 0x7f6e9806ef70 con 0x7f6e981422b0 2024-03-29T10:42:44.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 >> v1:172.21.15.136:6789/0 conn(0x7f6e980b17d0 legacy=0x7f6e9812ebb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:42:44.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.749+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 >> v1:172.21.15.117:6789/0 conn(0x7f6e98142690 legacy=0x7f6e981e6d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:44.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.750+0000 7f6e957fa640 10 monclient: _finish_hunting 0 2024-03-29T10:42:44.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.750+0000 7f6e957fa640 1 monclient: found mon.c 2024-03-29T10:42:44.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.750+0000 7f6e957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:44.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.750+0000 7f6e957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.750+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98150f10 con 0x7f6e981422b0 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.750+0000 7f6e957fa640 10 monclient: _finish_auth 0 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e957fa640 10 monclient: _check_auth_tickets 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e9eb90640 5 monclient: authenticate success, global_id 4577 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e9eb90640 10 monclient: _renew_subs 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e9eb90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e98150960 con 0x7f6e981422b0 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e9eb90640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e9eb90640 10 monclient: _renew_subs 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e9eb90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e980b69f0 con 0x7f6e981422b0 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e9eb90640 1 librados: init done 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e84001cd0 con 0x7f6e981422b0 2024-03-29T10:42:44.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.751+0000 7f6e96ffd640 4 set_mon_vals no callback set 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.752+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6e84003a60 con 0x7f6e981422b0 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.752+0000 7f6e957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.752+0000 7f6e957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.752+0000 7f6e957fa640 10 monclient: dump: 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:44.738 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:44.739 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:44.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.753+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f6e84003c40 con 0x7f6e981422b0 2024-03-29T10:42:44.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.754+0000 7f6e9eb90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:44.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.754+0000 7f6e9eb90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:44.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.754+0000 7f6e9eb90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:44.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.754+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e78005080 con 0x7f6e981422b0 2024-03-29T10:42:44.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.756+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f6e84084ba0 con 0x7f6e981422b0 2024-03-29T10:42:44.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.756+0000 7f6e957fa640 10 client.4577.objecter ms_dispatch 0x7f6e98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:44.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.756+0000 7f6e957fa640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:44.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.756+0000 7f6e957fa640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:44.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.756+0000 7f6e957fa640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T10:42:44.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.757+0000 7f6e9c92c640 10 monclient: get_auth_request con 0x7f6e7c06dc30 auth_method 0 2024-03-29T10:42:44.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.757+0000 7f6e957fa640 10 client.4577.objecter ms_handle_connect 0x7f6e7c06dc30 2024-03-29T10:42:44.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.764+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e84050480 con 0x7f6e981422b0 2024-03-29T10:42:44.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.764+0000 7f6e957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:44.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:44.764+0000 7f6e957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:45.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.086+0000 7f6e9eb90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:45.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.086+0000 7f6e9eb90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:45.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.086+0000 7f6e9eb90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:42:45.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.086+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e78005330 con 0x7f6e981422b0 2024-03-29T10:42:45.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.086+0000 7f6e957fa640 1 -- 172.21.15.117:0/312112508 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f6e840523b0 con 0x7f6e981422b0 2024-03-29T10:42:45.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.087+0000 7f6e957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:45.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.087+0000 7f6e957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:45.070 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:42:45.071 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:45.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.092+0000 7f6e9eb90640 10 librados: watch_flush enter 2024-03-29T10:42:45.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.093+0000 7f6e9eb90640 10 librados: watch_flush exit 2024-03-29T10:42:45.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.093+0000 7f6e9eb90640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T10:42:45.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.093+0000 7f6e9eb90640 10 client.4577.objecter successfully canceled tick 2024-03-29T10:42:45.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.093+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 >> v1:172.21.15.117:6816/376768171 conn(0x7f6e7c06dc30 legacy=0x7f6e7c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:45.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.093+0000 7f6e9eb90640 10 monclient: shutdown 2024-03-29T10:42:45.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.093+0000 7f6e9eb90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:45.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.093+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 >> v1:172.21.15.117:6790/0 conn(0x7f6e981422b0 legacy=0x7f6e9814f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:45.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.094+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 shutdown_connections 2024-03-29T10:42:45.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.094+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 >> 172.21.15.117:0/312112508 conn(0x7f6e98066a00 msgr2=0x7f6e9813f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:45.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.095+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 shutdown_connections 2024-03-29T10:42:45.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.095+0000 7f6e9eb90640 1 -- 172.21.15.117:0/312112508 wait complete. 2024-03-29T10:42:45.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:45.097+0000 7f6e9eb90640 1 librados: shutdown 2024-03-29T10:42:45.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:51.906 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:52.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.074+0000 7ff400d3e640 10 monclient: get_monmap_and_config 2024-03-29T10:42:52.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.074+0000 7ff400d3e640 10 monclient: build_initial_monmap 2024-03-29T10:42:52.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.074+0000 7ff400d3e640 10 monclient: monmap: 2024-03-29T10:42:52.058 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:52.058 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:52.058 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:42:52.075676+0000 2024-03-29T10:42:52.058 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:42:52.075676+0000 2024-03-29T10:42:52.059 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:52.059 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:52.059 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:52.059 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:42:52.059 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:42:52.059 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:52.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.075+0000 7ff400d3e640 1 Processor -- start 2024-03-29T10:42:52.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.076+0000 7ff400d3e640 1 -- start start 2024-03-29T10:42:52.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.076+0000 7ff400d3e640 10 monclient: init 2024-03-29T10:42:52.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3fc0b0860 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3fc072450 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3fc13ee40 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0ad840 con 0x7ff3fc0b0860 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc13c830 con 0x7ff3fc072450 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc13c9c0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff400d3e640 10 monclient(hunting): authenticate will time out at 1293.339558s 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff3fa59c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7ff3fc0b0860 0x7ff3fc0acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:39082/0 (socket says 172.21.15.117:39082) 2024-03-29T10:42:52.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.077+0000 7ff3fa59c640 1 -- 172.21.15.117:0/860730344 learned_addr learned my addr 172.21.15.117:0/860730344 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:52.062 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777184078 0 0) 0x7ff3fc13c830 con 0x7ff3fc072450 2024-03-29T10:42:52.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:52.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:52.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:52.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:42:52.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e4003520 con 0x7ff3fc072450 2024-03-29T10:42:52.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611193405 0 0) 0x7ff3fc13c9c0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:52.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc13c830 con 0x7ff3fc13ee40 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378646259 0 0) 0x7ff3e4003520 con 0x7ff3fc072450 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 >> v1:172.21.15.117:6790/0 conn(0x7ff3fc13ee40 legacy=0x7ff3fc13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 >> v1:172.21.15.117:6789/0 conn(0x7ff3fc0b0860 legacy=0x7ff3fc0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 monclient: found mon.noname-c 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:52.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc13f970 con 0x7ff3fc072450 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient: _finish_auth 0 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff3f0001690 con 0x7ff3fc072450 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.078+0000 7ff3f959a640 10 monclient: dump: 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:52.065 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:52.066 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:52.066 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:52.066 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:52.066 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:52.066 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:52.066 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:52.066 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:52.066 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:52.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff400d3e640 5 monclient: authenticate success, global_id 4486 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff400d3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f00017f0 con 0x7ff3fc072450 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff3f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff3f959a640 1 -- 172.21.15.117:0/860730344 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff3f0001c40 con 0x7ff3fc072450 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff3f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff3f959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff3f959a640 10 monclient: dump: 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:52.067 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:52.068 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:52.068 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:52.068 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:52.068 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:52.068 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:52.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff400d3e640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:52.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.079+0000 7ff400d3e640 4 set_mon_vals no callback set 2024-03-29T10:42:52.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.080+0000 7ff400d3e640 10 monclient: shutdown 2024-03-29T10:42:52.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.080+0000 7ff400d3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:52.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.080+0000 7ff400d3e640 1 -- 172.21.15.117:0/860730344 >> v1:172.21.15.136:6789/0 conn(0x7ff3fc072450 legacy=0x7ff3fc13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:52.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.081+0000 7ff400d3e640 1 -- 172.21.15.117:0/860730344 shutdown_connections 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.081+0000 7ff400d3e640 1 -- 172.21.15.117:0/860730344 >> 172.21.15.117:0/860730344 conn(0x7ff3fc06b660 msgr2=0x7ff3fc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.081+0000 7ff400d3e640 1 -- 172.21.15.117:0/860730344 shutdown_connections 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.081+0000 7ff400d3e640 1 -- 172.21.15.117:0/860730344 wait complete. 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.082+0000 7ff400d3e640 10 monclient: build_initial_monmap 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.082+0000 7ff400d3e640 10 monclient: monmap: 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:52.079 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.082+0000 7ff400d3e640 1 librados: starting msgr at 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.082+0000 7ff400d3e640 1 librados: starting objecter 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.083+0000 7ff400d3e640 1 Processor -- start 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.083+0000 7ff400d3e640 1 -- start start 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.083+0000 7ff400d3e640 1 librados: setting wanted keys 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.083+0000 7ff400d3e640 1 librados: calling monclient init 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.083+0000 7ff400d3e640 10 monclient: init 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:52.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3fc072450 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3fc0b0860 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3fc13ee40 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0b2b80 con 0x7ff3fc0b0860 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0b1590 con 0x7ff3fc13ee40 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0b11d0 con 0x7ff3fc072450 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff400d3e640 10 monclient(hunting): authenticate will time out at 1293.346271s 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff3fa59c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ff3fc072450 0x7ff3fc12e950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:57510/0 (socket says 172.21.15.117:57510) 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff3fa59c640 1 -- 172.21.15.117:0/2936748004 learned_addr learned my addr 172.21.15.117:0/2936748004 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff3eaffd640 10 client.?.objecter ms_handle_connect 0x7ff3fc072450 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff3eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:52.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff3eaffd640 10 client.?.objecter ms_handle_connect 0x7ff3fc13ee40 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.084+0000 7ff3eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668055115 0 0) 0x7ff3fc0b1590 con 0x7ff3fc13ee40 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc004eb0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 client.?.objecter ms_handle_connect 0x7ff3fc0b0860 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822446262 0 0) 0x7ff3fc0b11d0 con 0x7ff3fc072450 2024-03-29T10:42:52.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): my global_id is 4583 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc0b1590 con 0x7ff3fc072450 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2474879835 0 0) 0x7ff3dc004eb0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3fc0b11d0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff3f4001a80 con 0x7ff3fc13ee40 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): dump: 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:52.083 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.085+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519247617 0 0) 0x7ff3fc0b2b80 con 0x7ff3fc0b0860 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:52.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc004eb0 con 0x7ff3fc0b0860 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 632320536 0 0) 0x7ff3fc0b11d0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 >> v1:172.21.15.117:6790/0 conn(0x7ff3fc072450 legacy=0x7ff3fc12e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 >> v1:172.21.15.117:6789/0 conn(0x7ff3fc0b0860 legacy=0x7ff3fc12f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 1 monclient: found mon.b 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc0b1360 con 0x7ff3fc13ee40 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:52.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff3eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff400d3e640 5 monclient: authenticate success, global_id 4489 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff400d3e640 10 monclient: _renew_subs 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff400d3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc12f060 con 0x7ff3fc13ee40 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff400d3e640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff400d3e640 10 monclient: _renew_subs 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff400d3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc13e300 con 0x7ff3fc13ee40 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.086+0000 7ff400d3e640 1 librados: init done 2024-03-29T10:42:52.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.087+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f4001be0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.087+0000 7ff3eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.087+0000 7ff3f8d99640 4 set_mon_vals no callback set 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.087+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ff3f4003970 con 0x7ff3fc13ee40 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.087+0000 7ff3eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.087+0000 7ff3eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.088+0000 7ff400d3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.087+0000 7ff3eaffd640 10 monclient: dump: 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:52.087 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.088+0000 7ff400d3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.088+0000 7ff400d3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.088+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e0005080 con 0x7ff3fc13ee40 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.088+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7ff3f4019a00 con 0x7ff3fc13ee40 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.089+0000 7ff3fa59c640 10 monclient: get_auth_request con 0x7ff3dc06de50 auth_method 0 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.090+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7ff3f40839c0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.090+0000 7ff3eaffd640 10 client.4489.objecter ms_dispatch 0x7ff3fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:52.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.090+0000 7ff3eaffd640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:52.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.090+0000 7ff3eaffd640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:52.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.090+0000 7ff3eaffd640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-29T10:42:52.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.090+0000 7ff3eaffd640 10 client.4489.objecter ms_handle_connect 0x7ff3dc06de50 2024-03-29T10:42:52.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.096+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3f404f0d0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.096+0000 7ff3eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:52.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.096+0000 7ff3eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:52.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.486+0000 7ff400d3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:52.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.486+0000 7ff400d3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:52.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.486+0000 7ff400d3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:42:52.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.486+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3e0005330 con 0x7ff3fc13ee40 2024-03-29T10:42:52.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.487+0000 7ff3eaffd640 1 -- 172.21.15.117:0/2936748004 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7ff3f40511d0 con 0x7ff3fc13ee40 2024-03-29T10:42:52.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.487+0000 7ff3eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:52.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.487+0000 7ff3eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:52.471 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:42:52.471 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:52.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 10 librados: watch_flush enter 2024-03-29T10:42:52.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 10 librados: watch_flush exit 2024-03-29T10:42:52.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-29T10:42:52.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 10 client.4489.objecter successfully canceled tick 2024-03-29T10:42:52.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 >> v1:172.21.15.117:6816/376768171 conn(0x7ff3dc06de50 legacy=0x7ff3dc070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:52.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 10 monclient: shutdown 2024-03-29T10:42:52.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:52.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 >> v1:172.21.15.136:6789/0 conn(0x7ff3fc13ee40 legacy=0x7ff3fc12fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:52.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 shutdown_connections 2024-03-29T10:42:52.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 >> 172.21.15.117:0/2936748004 conn(0x7ff3fc06b660 msgr2=0x7ff3fc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:52.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 shutdown_connections 2024-03-29T10:42:52.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 1 -- 172.21.15.117:0/2936748004 wait complete. 2024-03-29T10:42:52.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:52.490+0000 7ff400d3e640 1 librados: shutdown 2024-03-29T10:42:52.487 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:59.295 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.454+0000 7f1a37fb5640 10 monclient: get_monmap_and_config 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.454+0000 7f1a37fb5640 10 monclient: build_initial_monmap 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.454+0000 7f1a37fb5640 10 monclient: monmap: 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:42:59.455535+0000 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:42:59.455535+0000 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:59.438 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:42:59.439 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:42:59.439 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.455+0000 7f1a37fb5640 1 Processor -- start 2024-03-29T10:42:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.455+0000 7f1a37fb5640 1 -- start start 2024-03-29T10:42:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.455+0000 7f1a37fb5640 10 monclient: init 2024-03-29T10:42:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f1a301422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:59.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f1a30142690 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f1a300b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a30142c80 con 0x7f1a30142690 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300aae40 con 0x7f1a301422b0 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3013fbd0 con 0x7f1a300b17d0 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a37fb5640 10 monclient(hunting): authenticate will time out at 1300.718873s 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a35550640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1a30142690 0x7f1a300b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:54910/0 (socket says 172.21.15.117:54910) 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.456+0000 7f1a35550640 1 -- 172.21.15.117:0/1980073630 learned_addr learned my addr 172.21.15.117:0/1980073630 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.457+0000 7f1a34d4f640 1 -- 172.21.15.117:0/1980073630 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376070341 0 0) 0x7f1a30142c80 con 0x7f1a30142690 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.457+0000 7f1a34d4f640 10 monclient(hunting): _negotiate 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.457+0000 7f1a34d4f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.457+0000 7f1a34d4f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.457+0000 7f1a34d4f640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:42:59.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.457+0000 7f1a34d4f640 1 -- 172.21.15.117:0/1980073630 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a1c003520 con 0x7f1a30142690 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.457+0000 7f1a34d4f640 1 -- 172.21.15.117:0/1980073630 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 113751855 0 0) 0x7f1a1c003520 con 0x7f1a30142690 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.457+0000 7f1a34d4f640 1 -- 172.21.15.117:0/1980073630 >> v1:172.21.15.117:6790/0 conn(0x7f1a300b17d0 legacy=0x7f1a300b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 1 -- 172.21.15.117:0/1980073630 >> v1:172.21.15.136:6789/0 conn(0x7f1a301422b0 legacy=0x7f1a3013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: _finish_hunting 0 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 1 monclient: found mon.noname-a 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 1 -- 172.21.15.117:0/1980073630 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a30151940 con 0x7f1a30142690 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: _finish_auth 0 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: _check_auth_tickets 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 1 -- 172.21.15.117:0/1980073630 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1a20001670 con 0x7f1a30142690 2024-03-29T10:42:59.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: dump: 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a37fb5640 5 monclient: authenticate success, global_id 4524 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a37fb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:59.447 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 1 -- 172.21.15.117:0/1980073630 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a20001890 con 0x7f1a30142690 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 1 -- 172.21.15.117:0/1980073630 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1a20001c60 con 0x7f1a30142690 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a34d4f640 10 monclient: dump: 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:59.448 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a37fb5640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.458+0000 7f1a37fb5640 4 set_mon_vals no callback set 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.459+0000 7f1a37fb5640 10 monclient: shutdown 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.460+0000 7f1a37fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.460+0000 7f1a37fb5640 1 -- 172.21.15.117:0/1980073630 >> v1:172.21.15.117:6789/0 conn(0x7f1a30142690 legacy=0x7f1a300b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.460+0000 7f1a37fb5640 1 -- 172.21.15.117:0/1980073630 shutdown_connections 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.460+0000 7f1a37fb5640 1 -- 172.21.15.117:0/1980073630 >> 172.21.15.117:0/1980073630 conn(0x7f1a30066a00 msgr2=0x7f1a30066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.460+0000 7f1a37fb5640 1 -- 172.21.15.117:0/1980073630 shutdown_connections 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.460+0000 7f1a37fb5640 1 -- 172.21.15.117:0/1980073630 wait complete. 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.461+0000 7f1a37fb5640 10 monclient: build_initial_monmap 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.461+0000 7f1a37fb5640 10 monclient: monmap: 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:42:59.449 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:42:59.450 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:59.450 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:59.450 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:42:59.450 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:42:59.450 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:42:59.450 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:59.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.461+0000 7f1a37fb5640 1 librados: starting msgr at 2024-03-29T10:42:59.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.461+0000 7f1a37fb5640 1 librados: starting objecter 2024-03-29T10:42:59.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.462+0000 7f1a37fb5640 1 Processor -- start 2024-03-29T10:42:59.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.462+0000 7f1a37fb5640 1 -- start start 2024-03-29T10:42:59.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.462+0000 7f1a37fb5640 1 librados: setting wanted keys 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.462+0000 7f1a37fb5640 1 librados: calling monclient init 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.462+0000 7f1a37fb5640 10 monclient: init 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.462+0000 7f1a37fb5640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a37fb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a37fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f1a300b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a37fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f1a301422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a37fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f1a30142690 addr v1:172.21.15.117:6789/0 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a37fb5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300b7930 con 0x7f1a30142690 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a37fb5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300b6cc0 con 0x7f1a301422b0 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a37fb5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300b6fa0 con 0x7f1a300b17d0 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a37fb5640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a37fb5640 10 monclient(hunting): authenticate will time out at 1300.725236s 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a36552640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1a30142690 0x7f1a30140550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:55256/0 (socket says 172.21.15.117:55256) 2024-03-29T10:42:59.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a36552640 1 -- 172.21.15.117:0/4078550439 learned_addr learned my addr 172.21.15.117:0/4078550439 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 client.?.objecter ms_handle_connect 0x7f1a30142690 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 client.?.objecter ms_handle_connect 0x7f1a300b17d0 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 client.?.objecter ms_handle_connect 0x7f1a301422b0 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810525092 0 0) 0x7f1a300b7930 con 0x7f1a30142690 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.463+0000 7f1a267fc640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a14004eb0 con 0x7f1a30142690 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254447855 0 0) 0x7f1a300b6fa0 con 0x7f1a300b17d0 2024-03-29T10:42:59.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a300b7930 con 0x7f1a300b17d0 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1787136342 0 0) 0x7f1a14004eb0 con 0x7f1a30142690 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a300b6fa0 con 0x7f1a30142690 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1a28001c40 con 0x7f1a30142690 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): dump: 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.461 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767204030 0 0) 0x7f1a300b6cc0 con 0x7f1a301422b0 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a14004eb0 con 0x7f1a301422b0 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2930514500 0 0) 0x7f1a300b7930 con 0x7f1a300b17d0 2024-03-29T10:42:59.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a14005ab0 con 0x7f1a300b17d0 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1a2c0018b0 con 0x7f1a300b17d0 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.464+0000 7f1a267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient(hunting): dump: 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:59.463 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3505860851 0 0) 0x7f1a300b6fa0 con 0x7f1a30142690 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 >> v1:172.21.15.117:6790/0 conn(0x7f1a300b17d0 legacy=0x7f1a301502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 >> v1:172.21.15.136:6789/0 conn(0x7f1a301422b0 legacy=0x7f1a3013fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 1 monclient: found mon.a 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a301509d0 con 0x7f1a30142690 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a28003970 con 0x7f1a30142690 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a37fb5640 5 monclient: authenticate success, global_id 4527 2024-03-29T10:42:59.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a37fb5640 10 monclient: _renew_subs 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a37fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a300b73c0 con 0x7f1a30142690 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a37fb5640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a37fb5640 10 monclient: _renew_subs 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a37fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a300b12a0 con 0x7f1a30142690 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a37fb5640 1 librados: init done 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f1a28003d40 con 0x7f1a30142690 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a27fff640 4 set_mon_vals no callback set 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.465+0000 7f1a267fc640 10 monclient: dump: 2024-03-29T10:42:59.465 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.467+0000 7f1a37fb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.467+0000 7f1a37fb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.467+0000 7f1a37fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.467+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a300b75b0 con 0x7f1a30142690 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.467+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f1a28003ea0 con 0x7f1a30142690 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.468+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f1a28084d60 con 0x7f1a30142690 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.468+0000 7f1a267fc640 10 client.4527.objecter ms_dispatch 0x7f1a30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.468+0000 7f1a267fc640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.468+0000 7f1a267fc640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.468+0000 7f1a267fc640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.468+0000 7f1a35d51640 10 monclient: get_auth_request con 0x7f1a1406dfe0 auth_method 0 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.468+0000 7f1a267fc640 10 client.4527.objecter ms_handle_connect 0x7f1a1406dfe0 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.474+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a28050640 con 0x7f1a30142690 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.474+0000 7f1a267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.474+0000 7f1a267fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:59.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.836+0000 7f1a37fb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:59.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.836+0000 7f1a37fb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:59.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.836+0000 7f1a37fb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:42:59.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.836+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a30075f10 con 0x7f1a30142690 2024-03-29T10:42:59.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.838+0000 7f1a267fc640 1 -- 172.21.15.117:0/4078550439 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f1a28052570 con 0x7f1a30142690 2024-03-29T10:42:59.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.838+0000 7f1a267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:59.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.838+0000 7f1a267fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:59.821 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:42:59.821 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.841+0000 7f1a37fb5640 10 librados: watch_flush enter 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.841+0000 7f1a37fb5640 10 librados: watch_flush exit 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.841+0000 7f1a37fb5640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.841+0000 7f1a37fb5640 10 client.4527.objecter successfully canceled tick 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.842+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 >> v1:172.21.15.117:6816/376768171 conn(0x7f1a1406dfe0 legacy=0x7f1a140704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.842+0000 7f1a37fb5640 10 monclient: shutdown 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.842+0000 7f1a37fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.842+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 >> v1:172.21.15.117:6789/0 conn(0x7f1a30142690 legacy=0x7f1a30140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.842+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 shutdown_connections 2024-03-29T10:42:59.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.842+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 >> 172.21.15.117:0/4078550439 conn(0x7f1a30066a00 msgr2=0x7f1a300b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:59.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.842+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 shutdown_connections 2024-03-29T10:42:59.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.842+0000 7f1a37fb5640 1 -- 172.21.15.117:0/4078550439 wait complete. 2024-03-29T10:42:59.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:42:59.842+0000 7f1a37fb5640 1 librados: shutdown 2024-03-29T10:42:59.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:06.646 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:06.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.806+0000 7f614cde2640 10 monclient: get_monmap_and_config 2024-03-29T10:43:06.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.806+0000 7f614cde2640 10 monclient: build_initial_monmap 2024-03-29T10:43:06.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.806+0000 7f614cde2640 10 monclient: monmap: 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:43:06.807316+0000 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:43:06.807316+0000 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:43:06.790 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.807+0000 7f614cde2640 1 Processor -- start 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.807+0000 7f614cde2640 1 -- start start 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.807+0000 7f614cde2640 10 monclient: init 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 10 monclient(hunting): picked mon.noname-b con 0x7f61480b4330 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 10 monclient(hunting): picked mon.noname-c con 0x7f61481470b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 10 monclient(hunting): picked mon.noname-a con 0x7f61481439b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614806f2a0 con 0x7f61481439b0 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6148146b90 con 0x7f61481470b0 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61480b2ca0 con 0x7f61480b4330 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:06.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.808+0000 7f614cde2640 10 monclient(hunting): authenticate will time out at 1308.070855s 2024-03-29T10:43:06.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.810+0000 7f6146d9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f61481439b0 0x7f6148151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48566/0 (socket says 172.21.15.117:48566) 2024-03-29T10:43:06.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.810+0000 7f6146d9d640 1 -- 172.21.15.117:0/3341574446 learned_addr learned my addr 172.21.15.117:0/3341574446 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:06.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.810+0000 7f614559a640 1 -- 172.21.15.117:0/3341574446 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998683167 0 0) 0x7f614806f2a0 con 0x7f61481439b0 2024-03-29T10:43:06.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.810+0000 7f614559a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.810+0000 7f614559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.810+0000 7f614559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.810+0000 7f614559a640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:43:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.810+0000 7f614559a640 1 -- 172.21.15.117:0/3341574446 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6130003520 con 0x7f61481439b0 2024-03-29T10:43:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.811+0000 7f614559a640 1 -- 172.21.15.117:0/3341574446 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2195889120 0 0) 0x7f6130003520 con 0x7f61481439b0 2024-03-29T10:43:06.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.811+0000 7f614559a640 1 -- 172.21.15.117:0/3341574446 >> v1:172.21.15.117:6790/0 conn(0x7f61480b4330 legacy=0x7f61480b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 1 -- 172.21.15.117:0/3341574446 >> v1:172.21.15.136:6789/0 conn(0x7f61481470b0 legacy=0x7f61480b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 10 monclient: _finish_hunting 0 2024-03-29T10:43:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 1 monclient: found mon.noname-a 2024-03-29T10:43:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:43:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 1 -- 172.21.15.117:0/3341574446 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614806e0c0 con 0x7f61481439b0 2024-03-29T10:43:06.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 10 monclient: _finish_auth 0 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 10 monclient: _check_auth_tickets 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 1 -- 172.21.15.117:0/3341574446 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6138001690 con 0x7f61481439b0 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614559a640 10 monclient: dump: 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:06.798 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614cde2640 5 monclient: authenticate success, global_id 4530 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.812+0000 7f614cde2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.814+0000 7f614559a640 1 -- 172.21.15.117:0/3341574446 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6138001860 con 0x7f61481439b0 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.814+0000 7f614559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.814+0000 7f614cde2640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.814+0000 7f614cde2640 4 set_mon_vals no callback set 2024-03-29T10:43:06.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.814+0000 7f614cde2640 10 monclient: shutdown 2024-03-29T10:43:06.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.814+0000 7f614cde2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:06.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.814+0000 7f614cde2640 1 -- 172.21.15.117:0/3341574446 >> v1:172.21.15.117:6789/0 conn(0x7f61481439b0 legacy=0x7f6148151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:06.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.815+0000 7f614cde2640 1 -- 172.21.15.117:0/3341574446 shutdown_connections 2024-03-29T10:43:06.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.815+0000 7f614cde2640 1 -- 172.21.15.117:0/3341574446 >> 172.21.15.117:0/3341574446 conn(0x7f6148066a00 msgr2=0x7f6148066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.815+0000 7f614cde2640 1 -- 172.21.15.117:0/3341574446 shutdown_connections 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.815+0000 7f614cde2640 1 -- 172.21.15.117:0/3341574446 wait complete. 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.816+0000 7f614cde2640 10 monclient: build_initial_monmap 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.816+0000 7f614cde2640 10 monclient: monmap: 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.816+0000 7f614cde2640 1 librados: starting msgr at 2024-03-29T10:43:06.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.816+0000 7f614cde2640 1 librados: starting objecter 2024-03-29T10:43:06.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.817+0000 7f614cde2640 1 Processor -- start 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.817+0000 7f614cde2640 1 -- start start 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.817+0000 7f614cde2640 1 librados: setting wanted keys 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.817+0000 7f614cde2640 1 librados: calling monclient init 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.817+0000 7f614cde2640 10 monclient: init 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 10 monclient(hunting): picked mon.noname-b con 0x7f61480b4330 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 10 monclient(hunting): picked mon.noname-c con 0x7f61481439b0 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 10 monclient(hunting): picked mon.noname-a con 0x7f61481470b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61481dc130 con 0x7f61481470b0 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61481dc2c0 con 0x7f61480b4330 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614811f770 con 0x7f61481439b0 2024-03-29T10:43:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.818+0000 7f614cde2640 10 monclient(hunting): authenticate will time out at 1308.080826s 2024-03-29T10:43:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.819+0000 7f6146d9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f61481470b0 0x7f61481e6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48834/0 (socket says 172.21.15.117:48834) 2024-03-29T10:43:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.819+0000 7f6146d9d640 1 -- 172.21.15.117:0/3954066722 learned_addr learned my addr 172.21.15.117:0/3954066722 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.819+0000 7f6136ffd640 10 client.?.objecter ms_handle_connect 0x7f61481470b0 2024-03-29T10:43:06.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.819+0000 7f6136ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 client.?.objecter ms_handle_connect 0x7f61480b4330 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384353497 0 0) 0x7f61481dc130 con 0x7f61481470b0 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6128004eb0 con 0x7f61481470b0 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020777511 0 0) 0x7f61481dc2c0 con 0x7f61480b4330 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:06.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61481dc130 con 0x7f61480b4330 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.822+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 930622250 0 0) 0x7f6128004eb0 con 0x7f61481470b0 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61481dc2c0 con 0x7f61481470b0 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6138004980 con 0x7f61481470b0 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:06.812 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2910928194 0 0) 0x7f61481dc130 con 0x7f61480b4330 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6128004eb0 con 0x7f61480b4330 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6140001a90 con 0x7f61480b4330 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:06.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 924149363 0 0) 0x7f61481dc2c0 con 0x7f61481470b0 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 >> v1:172.21.15.117:6790/0 conn(0x7f61481439b0 legacy=0x7f61481401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 >> v1:172.21.15.136:6789/0 conn(0x7f61480b4330 legacy=0x7f61481dba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:06.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 monclient: found mon.a 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614811f900 con 0x7f61481470b0 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f614cde2640 5 monclient: authenticate success, global_id 4533 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f614cde2640 10 monclient: _renew_subs 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f614cde2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f614cde2640 1 -- 172.21.15.117:0/3954066722 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6148141410 con 0x7f61481470b0 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 client.4533.objecter ms_handle_connect 0x7f61481439b0 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f6136ffd640 10 client.4533.objecter resend_mon_ops 2024-03-29T10:43:06.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f614cde2640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f614cde2640 10 monclient: _renew_subs 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f614cde2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.823+0000 7f614cde2640 1 -- 172.21.15.117:0/3954066722 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61480b7120 con 0x7f61481470b0 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.824+0000 7f614cde2640 1 librados: init done 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.824+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6138001b10 con 0x7f61481470b0 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.824+0000 7f6136ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.824+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6138004f30 con 0x7f61481470b0 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.824+0000 7f6136ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.824+0000 7f6136ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.824+0000 7f6137fff640 4 set_mon_vals no callback set 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.825+0000 7f614cde2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.827+0000 7f6136ffd640 10 monclient: dump: 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:06.816 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:06.817 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:06.817 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:06.817 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:06.817 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:06.817 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:06.817 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:06.817 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:06.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.827+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f6138005090 con 0x7f61481470b0 2024-03-29T10:43:06.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.828+0000 7f614cde2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:06.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.828+0000 7f614cde2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:43:06.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.828+0000 7f614cde2640 1 -- 172.21.15.117:0/3954066722 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c005080 con 0x7f61481470b0 2024-03-29T10:43:06.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.828+0000 7f614659c640 10 monclient: get_auth_request con 0x7f612806de30 auth_method 0 2024-03-29T10:43:06.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.828+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f6138084c60 con 0x7f61481470b0 2024-03-29T10:43:06.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.828+0000 7f6136ffd640 10 client.4533.objecter ms_dispatch 0x7f6148000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:06.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.829+0000 7f6136ffd640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:06.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.829+0000 7f6136ffd640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:06.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.829+0000 7f6136ffd640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T10:43:06.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.829+0000 7f6136ffd640 10 client.4533.objecter ms_handle_connect 0x7f612806de30 2024-03-29T10:43:06.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.836+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6138050540 con 0x7f61481470b0 2024-03-29T10:43:06.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.836+0000 7f6136ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:06.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:06.836+0000 7f6136ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:07.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.161+0000 7f614cde2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:07.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.161+0000 7f614cde2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:07.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.161+0000 7f614cde2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:43:07.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.161+0000 7f614cde2640 1 -- 172.21.15.117:0/3954066722 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f612c005330 con 0x7f61481470b0 2024-03-29T10:43:07.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.163+0000 7f6136ffd640 1 -- 172.21.15.117:0/3954066722 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f6138052470 con 0x7f61481470b0 2024-03-29T10:43:07.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.163+0000 7f6136ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:07.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.163+0000 7f6136ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:07.147 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:43:07.147 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:07.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.167+0000 7f6134ff9640 10 librados: watch_flush enter 2024-03-29T10:43:07.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.167+0000 7f6134ff9640 10 librados: watch_flush exit 2024-03-29T10:43:07.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.167+0000 7f6134ff9640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T10:43:07.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.167+0000 7f6134ff9640 10 client.4533.objecter successfully canceled tick 2024-03-29T10:43:07.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.168+0000 7f6134ff9640 1 -- 172.21.15.117:0/3954066722 >> v1:172.21.15.117:6816/376768171 conn(0x7f612806de30 legacy=0x7f61280702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:07.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.168+0000 7f6134ff9640 10 monclient: shutdown 2024-03-29T10:43:07.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.168+0000 7f6134ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.168+0000 7f6134ff9640 1 -- 172.21.15.117:0/3954066722 >> v1:172.21.15.117:6789/0 conn(0x7f61481470b0 legacy=0x7f61481e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.168+0000 7f6134ff9640 1 -- 172.21.15.117:0/3954066722 shutdown_connections 2024-03-29T10:43:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.168+0000 7f6134ff9640 1 -- 172.21.15.117:0/3954066722 >> 172.21.15.117:0/3954066722 conn(0x7f6148066a00 msgr2=0x7f61480b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.169+0000 7f6134ff9640 1 -- 172.21.15.117:0/3954066722 shutdown_connections 2024-03-29T10:43:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.169+0000 7f6134ff9640 1 -- 172.21.15.117:0/3954066722 wait complete. 2024-03-29T10:43:07.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:07.169+0000 7f6134ff9640 1 librados: shutdown 2024-03-29T10:43:07.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:13.979 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:14.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.141+0000 7f7c77051640 10 monclient: get_monmap_and_config 2024-03-29T10:43:14.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.141+0000 7f7c77051640 10 monclient: build_initial_monmap 2024-03-29T10:43:14.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.141+0000 7f7c77051640 10 monclient: monmap: 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:43:14.142607+0000 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:43:14.142607+0000 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:14.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.142+0000 7f7c77051640 1 Processor -- start 2024-03-29T10:43:14.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.142+0000 7f7c77051640 1 -- start start 2024-03-29T10:43:14.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.142+0000 7f7c77051640 10 monclient: init 2024-03-29T10:43:14.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.143+0000 7f7c77051640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:14.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.143+0000 7f7c77051640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:14.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.143+0000 7f7c77051640 10 monclient(hunting): picked mon.noname-a con 0x7f7c700a9750 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.143+0000 7f7c77051640 10 monclient(hunting): picked mon.noname-b con 0x7f7c700ae220 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.144+0000 7f7c77051640 10 monclient(hunting): picked mon.noname-c con 0x7f7c7013b9d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.144+0000 7f7c77051640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c70142490 con 0x7f7c700a9750 2024-03-29T10:43:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.144+0000 7f7c77051640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c700ac130 con 0x7f7c7013b9d0 2024-03-29T10:43:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.144+0000 7f7c77051640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c700ac410 con 0x7f7c700ae220 2024-03-29T10:43:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.144+0000 7f7c77051640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.144+0000 7f7c77051640 10 monclient(hunting): authenticate will time out at 1315.406288s 2024-03-29T10:43:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.144+0000 7f7c76850640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7c7013b9d0 0x7f7c70149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:37362/0 (socket says 172.21.15.117:37362) 2024-03-29T10:43:14.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.144+0000 7f7c76850640 1 -- 172.21.15.117:0/698284983 learned_addr learned my addr 172.21.15.117:0/698284983 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:14.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.146+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393564107 0 0) 0x7f7c700ac130 con 0x7f7c7013b9d0 2024-03-29T10:43:14.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.146+0000 7f7c7504d640 10 monclient(hunting): _negotiate 2024-03-29T10:43:14.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.146+0000 7f7c7504d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:14.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.146+0000 7f7c7504d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:14.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.146+0000 7f7c7504d640 10 monclient(hunting): my global_id is 4501 2024-03-29T10:43:14.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.146+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c5c003520 con 0x7f7c7013b9d0 2024-03-29T10:43:14.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.147+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252036805 0 0) 0x7f7c700ac410 con 0x7f7c700ae220 2024-03-29T10:43:14.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.147+0000 7f7c7504d640 10 monclient(hunting): _negotiate 2024-03-29T10:43:14.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.147+0000 7f7c7504d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:14.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.147+0000 7f7c7504d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:14.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.147+0000 7f7c7504d640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:43:14.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.147+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c700ac130 con 0x7f7c700ae220 2024-03-29T10:43:14.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.147+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4201769321 0 0) 0x7f7c700ac130 con 0x7f7c700ae220 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.147+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 >> v1:172.21.15.136:6789/0 conn(0x7f7c7013b9d0 legacy=0x7f7c70149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.147+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 >> v1:172.21.15.117:6789/0 conn(0x7f7c700a9750 legacy=0x7f7c70141d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: _finish_hunting 0 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 1 monclient: found mon.noname-b 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700afa40 con 0x7f7c700ae220 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: _finish_auth 0 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: _check_auth_tickets 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7c60001670 con 0x7f7c700ae220 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: dump: 2024-03-29T10:43:14.133 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c77051640 5 monclient: authenticate success, global_id 4592 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c77051640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:14.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c600017d0 con 0x7f7c700ae220 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 1 -- 172.21.15.117:0/698284983 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7c60001ba0 con 0x7f7c700ae220 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c7504d640 10 monclient: dump: 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:14.135 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c77051640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.148+0000 7f7c77051640 4 set_mon_vals no callback set 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.149+0000 7f7c77051640 10 monclient: shutdown 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.149+0000 7f7c77051640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.149+0000 7f7c77051640 1 -- 172.21.15.117:0/698284983 >> v1:172.21.15.117:6790/0 conn(0x7f7c700ae220 legacy=0x7f7c7013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.150+0000 7f7c77051640 1 -- 172.21.15.117:0/698284983 shutdown_connections 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.150+0000 7f7c77051640 1 -- 172.21.15.117:0/698284983 >> 172.21.15.117:0/698284983 conn(0x7f7c700673e0 msgr2=0x7f7c700b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.150+0000 7f7c77051640 1 -- 172.21.15.117:0/698284983 shutdown_connections 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.150+0000 7f7c77051640 1 -- 172.21.15.117:0/698284983 wait complete. 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.151+0000 7f7c77051640 10 monclient: build_initial_monmap 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.152+0000 7f7c77051640 10 monclient: monmap: 2024-03-29T10:43:14.138 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.152+0000 7f7c77051640 1 librados: starting msgr at 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.153+0000 7f7c77051640 1 librados: starting objecter 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.153+0000 7f7c77051640 1 Processor -- start 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.153+0000 7f7c77051640 1 -- start start 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.153+0000 7f7c77051640 1 librados: setting wanted keys 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.153+0000 7f7c77051640 1 librados: calling monclient init 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.153+0000 7f7c77051640 10 monclient: init 2024-03-29T10:43:14.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.154+0000 7f7c77051640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:14.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.154+0000 7f7c77051640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:14.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.156+0000 7f7c77051640 10 monclient(hunting): picked mon.noname-a con 0x7f7c700a9750 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:14.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c77051640 10 monclient(hunting): picked mon.noname-c con 0x7f7c700ae220 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:14.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c77051640 10 monclient(hunting): picked mon.noname-b con 0x7f7c7013b9d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:14.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c77051640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7013c740 con 0x7f7c700a9750 2024-03-29T10:43:14.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c7604f640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7c700a9750 0x7f7c70071990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:50971/0 (socket says 172.21.15.117:50971) 2024-03-29T10:43:14.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c7604f640 1 -- 172.21.15.117:0/4263171214 learned_addr learned my addr 172.21.15.117:0/4263171214 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:14.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c77051640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7013ca20 con 0x7f7c7013b9d0 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c77051640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7013be30 con 0x7f7c700ae220 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c77051640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c77051640 10 monclient(hunting): authenticate will time out at 1315.421889s 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.159+0000 7f7c66ffd640 10 client.?.objecter ms_handle_connect 0x7f7c700a9750 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 10 client.?.objecter ms_handle_connect 0x7f7c7013b9d0 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 10 client.?.objecter ms_handle_connect 0x7f7c700ae220 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554067449 0 0) 0x7f7c7013be30 con 0x7f7c700ae220 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:14.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 10 monclient(hunting): my global_id is 4595 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.160+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c54004eb0 con 0x7f7c700ae220 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644718047 0 0) 0x7f7c7013c740 con 0x7f7c700a9750 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): my global_id is 4536 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7013be30 con 0x7f7c700a9750 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763175017 0 0) 0x7f7c7013ca20 con 0x7f7c7013b9d0 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:14.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): my global_id is 4504 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7013c740 con 0x7f7c7013b9d0 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3228397535 0 0) 0x7f7c54004eb0 con 0x7f7c700ae220 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c7013ca20 con 0x7f7c700ae220 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7c60001ae0 con 0x7f7c700ae220 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:14.155 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1760328895 0 0) 0x7f7c7013be30 con 0x7f7c700a9750 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c54004eb0 con 0x7f7c700a9750 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7c6c0018e0 con 0x7f7c700a9750 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:14.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 385889479 0 0) 0x7f7c7013ca20 con 0x7f7c700ae220 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 >> v1:172.21.15.136:6789/0 conn(0x7f7c7013b9d0 legacy=0x7f7c7006f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 >> v1:172.21.15.117:6789/0 conn(0x7f7c700a9750 legacy=0x7f7c70071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 monclient: found mon.c 2024-03-29T10:43:14.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c70070da0 con 0x7f7c700ae220 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c77051640 5 monclient: authenticate success, global_id 4595 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c77051640 10 monclient: _renew_subs 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c77051640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.161+0000 7f7c77051640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c70070f90 con 0x7f7c700ae220 2024-03-29T10:43:14.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c77051640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c77051640 10 monclient: _renew_subs 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c77051640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c77051640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c701e0da0 con 0x7f7c700ae220 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c77051640 1 librados: init done 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c600038a0 con 0x7f7c700ae220 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7c60003bb0 con 0x7f7c700ae220 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c66ffd640 10 monclient: dump: 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:14.159 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:14.160 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:14.160 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:14.160 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:14.160 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:14.160 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.162+0000 7f7c7484c640 4 set_mon_vals no callback set 2024-03-29T10:43:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.164+0000 7f7c64ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.164+0000 7f7c64ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:14.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.164+0000 7f7c64ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.164+0000 7f7c64ff9640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4c005080 con 0x7f7c700ae220 2024-03-29T10:43:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.166+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f7c60019b80 con 0x7f7c700ae220 2024-03-29T10:43:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.167+0000 7f7c7604f640 10 monclient: get_auth_request con 0x7f7c5406dfb0 auth_method 0 2024-03-29T10:43:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.167+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f7c60083850 con 0x7f7c700ae220 2024-03-29T10:43:14.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.167+0000 7f7c66ffd640 10 client.4595.objecter ms_dispatch 0x7f7c70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:14.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.167+0000 7f7c66ffd640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:14.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.167+0000 7f7c66ffd640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:14.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.167+0000 7f7c66ffd640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T10:43:14.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.168+0000 7f7c66ffd640 10 client.4595.objecter ms_handle_connect 0x7f7c5406dfb0 2024-03-29T10:43:14.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.174+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c6004f130 con 0x7f7c700ae220 2024-03-29T10:43:14.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.174+0000 7f7c66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:14.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.174+0000 7f7c66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:14.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.519+0000 7f7c64ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:14.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.519+0000 7f7c64ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:14.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.519+0000 7f7c64ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:14.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.519+0000 7f7c64ff9640 1 -- 172.21.15.117:0/4263171214 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c4c005330 con 0x7f7c700ae220 2024-03-29T10:43:14.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.522+0000 7f7c66ffd640 1 -- 172.21.15.117:0/4263171214 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f7c60051060 con 0x7f7c700ae220 2024-03-29T10:43:14.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.522+0000 7f7c66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:14.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.522+0000 7f7c66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:14.506 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:43:14.506 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:14.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 10 librados: watch_flush enter 2024-03-29T10:43:14.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 10 librados: watch_flush exit 2024-03-29T10:43:14.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T10:43:14.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 10 client.4595.objecter successfully canceled tick 2024-03-29T10:43:14.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 1 -- 172.21.15.117:0/4263171214 >> v1:172.21.15.117:6816/376768171 conn(0x7f7c5406dfb0 legacy=0x7f7c54070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:14.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 10 monclient: shutdown 2024-03-29T10:43:14.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:14.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 1 -- 172.21.15.117:0/4263171214 >> v1:172.21.15.117:6790/0 conn(0x7f7c700ae220 legacy=0x7f7c7006e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:14.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 1 -- 172.21.15.117:0/4263171214 shutdown_connections 2024-03-29T10:43:14.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.526+0000 7f7c64ff9640 1 -- 172.21.15.117:0/4263171214 >> 172.21.15.117:0/4263171214 conn(0x7f7c700673e0 msgr2=0x7f7c70069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:14.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.527+0000 7f7c64ff9640 1 -- 172.21.15.117:0/4263171214 shutdown_connections 2024-03-29T10:43:14.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.527+0000 7f7c64ff9640 1 -- 172.21.15.117:0/4263171214 wait complete. 2024-03-29T10:43:14.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:14.527+0000 7f7c64ff9640 1 librados: shutdown 2024-03-29T10:43:14.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:21.332 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.496+0000 7f6881577640 10 monclient: get_monmap_and_config 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.496+0000 7f6881577640 10 monclient: build_initial_monmap 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.496+0000 7f6881577640 10 monclient: monmap: 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:43:21.496968+0000 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:43:21.496968+0000 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:43:21.480 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:21.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.496+0000 7f6881577640 1 Processor -- start 2024-03-29T10:43:21.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.497+0000 7f6881577640 1 -- start start 2024-03-29T10:43:21.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.497+0000 7f6881577640 10 monclient: init 2024-03-29T10:43:21.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.497+0000 7f6881577640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:21.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.497+0000 7f6881577640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:21.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.497+0000 7f6881577640 10 monclient(hunting): picked mon.noname-c con 0x7f687c0b1f60 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:21.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.497+0000 7f6881577640 10 monclient(hunting): picked mon.noname-a con 0x7f687c071eb0 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:21.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6881577640 10 monclient(hunting): picked mon.noname-b con 0x7f687c143d10 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:21.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6881577640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c06ef80 con 0x7f687c071eb0 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6881577640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c06f260 con 0x7f687c0b1f60 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6881577640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c0b40c0 con 0x7f687c143d10 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6881577640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6881577640 10 monclient(hunting): authenticate will time out at 1322.760133s 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f687b7fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f687c143d10 0x7f687c146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:38404/0 (socket says 172.21.15.117:38404) 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f687b7fe640 1 -- 172.21.15.117:0/501038784 learned_addr learned my addr 172.21.15.117:0/501038784 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6879ffb640 1 -- 172.21.15.117:0/501038784 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267214983 0 0) 0x7f687c0b40c0 con 0x7f687c143d10 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6879ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6879ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6879ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:21.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6879ffb640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:43:21.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.498+0000 7f6879ffb640 1 -- 172.21.15.117:0/501038784 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6864003520 con 0x7f687c143d10 2024-03-29T10:43:21.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 1 -- 172.21.15.117:0/501038784 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496437360 0 0) 0x7f6864003520 con 0x7f687c143d10 2024-03-29T10:43:21.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 1 -- 172.21.15.117:0/501038784 >> v1:172.21.15.136:6789/0 conn(0x7f687c0b1f60 legacy=0x7f687c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 1 -- 172.21.15.117:0/501038784 >> v1:172.21.15.117:6789/0 conn(0x7f687c071eb0 legacy=0x7f687c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 1 monclient: found mon.noname-b 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 1 -- 172.21.15.117:0/501038784 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c155d70 con 0x7f687c143d10 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.501+0000 7f6879ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6881577640 5 monclient: authenticate success, global_id 4598 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6881577640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6879ffb640 1 -- 172.21.15.117:0/501038784 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6874001a10 con 0x7f687c143d10 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6879ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6879ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:21.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6879ffb640 10 monclient: dump: 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6881577640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6879ffb640 1 -- 172.21.15.117:0/501038784 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6874001ca0 con 0x7f687c143d10 2024-03-29T10:43:21.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6879ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:21.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6881577640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:21.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.502+0000 7f6881577640 4 set_mon_vals no callback set 2024-03-29T10:43:21.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.503+0000 7f6881577640 10 monclient: shutdown 2024-03-29T10:43:21.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.503+0000 7f6881577640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:21.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.503+0000 7f6881577640 1 -- 172.21.15.117:0/501038784 >> v1:172.21.15.117:6790/0 conn(0x7f687c143d10 legacy=0x7f687c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:21.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.503+0000 7f6881577640 1 -- 172.21.15.117:0/501038784 shutdown_connections 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.503+0000 7f6881577640 1 -- 172.21.15.117:0/501038784 >> 172.21.15.117:0/501038784 conn(0x7f687c066a00 msgr2=0x7f687c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.504+0000 7f6881577640 1 -- 172.21.15.117:0/501038784 shutdown_connections 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.504+0000 7f6881577640 1 -- 172.21.15.117:0/501038784 wait complete. 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.505+0000 7f6881577640 10 monclient: build_initial_monmap 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.505+0000 7f6881577640 10 monclient: monmap: 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:43:21.490 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:43:21.491 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:21.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.505+0000 7f6881577640 1 librados: starting msgr at 2024-03-29T10:43:21.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.505+0000 7f6881577640 1 librados: starting objecter 2024-03-29T10:43:21.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.506+0000 7f6881577640 1 Processor -- start 2024-03-29T10:43:21.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.506+0000 7f6881577640 1 -- start start 2024-03-29T10:43:21.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.506+0000 7f6881577640 1 librados: setting wanted keys 2024-03-29T10:43:21.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.506+0000 7f6881577640 1 librados: calling monclient init 2024-03-29T10:43:21.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.506+0000 7f6881577640 10 monclient: init 2024-03-29T10:43:21.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:21.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:21.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 10 monclient(hunting): picked mon.noname-b con 0x7f687c0b1f60 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:21.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 10 monclient(hunting): picked mon.noname-a con 0x7f687c1542d0 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 10 monclient(hunting): picked mon.noname-c con 0x7f687c0b7bb0 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c0b87e0 con 0x7f687c1542d0 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c139550 con 0x7f687c0b1f60 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c0b7690 con 0x7f687c0b7bb0 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.507+0000 7f6881577640 10 monclient(hunting): authenticate will time out at 1322.769326s 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.510+0000 7f687b7fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f687c0b7bb0 0x7f687c1e6eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41610/0 (socket says 172.21.15.117:41610) 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.510+0000 7f687b7fe640 1 -- 172.21.15.117:0/607513960 learned_addr learned my addr 172.21.15.117:0/607513960 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 10 client.?.objecter ms_handle_connect 0x7f687c0b7bb0 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199500113 0 0) 0x7f687c0b7690 con 0x7f687c0b7bb0 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:21.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6868004eb0 con 0x7f687c0b7bb0 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 10 client.?.objecter ms_handle_connect 0x7f687c0b1f60 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.511+0000 7f68637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 436288876 0 0) 0x7f6868004eb0 con 0x7f687c0b7bb0 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f687c0b7690 con 0x7f687c0b7bb0 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6874004b60 con 0x7f687c0b7bb0 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 10 monclient(hunting): dump: 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:21.498 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2635345162 0 0) 0x7f687c0b7690 con 0x7f687c0b7bb0 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 >> v1:172.21.15.136:6789/0 conn(0x7f687c0b1f60 legacy=0x7f687c0b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.512+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 >> v1:172.21.15.117:6789/0 conn(0x7f687c1542d0 legacy=0x7f687c12fc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 1 monclient: found mon.c 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c0b8a30 con 0x7f687c0b7bb0 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:21.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:21.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:21.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6874004f70 con 0x7f687c0b7bb0 2024-03-29T10:43:21.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.513+0000 7f68637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:21.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f6874005380 con 0x7f687c0b7bb0 2024-03-29T10:43:21.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f68637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f68637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f68637fe640 10 monclient: dump: 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f6881577640 5 monclient: authenticate success, global_id 4601 2024-03-29T10:43:21.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f6881577640 10 monclient: _renew_subs 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f6881577640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687c0b5f20 con 0x7f687c0b7bb0 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f6881577640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f6881577640 10 monclient: _renew_subs 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f6881577640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f687c0b71e0 con 0x7f687c0b7bb0 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.514+0000 7f6881577640 1 librados: init done 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.515+0000 7f68797fa640 4 set_mon_vals no callback set 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.517+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f6874001e90 con 0x7f687c0b7bb0 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.517+0000 7f6881577640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.517+0000 7f6881577640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.517+0000 7f6881577640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.517+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685c005080 con 0x7f687c0b7bb0 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.518+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f6874084eb0 con 0x7f687c0b7bb0 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.518+0000 7f68637fe640 10 client.4601.objecter ms_dispatch 0x7f687c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:21.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.518+0000 7f68637fe640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.518+0000 7f68637fe640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.518+0000 7f68637fe640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T10:43:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.519+0000 7f687affd640 10 monclient: get_auth_request con 0x7f686806dbd0 auth_method 0 2024-03-29T10:43:21.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.519+0000 7f68637fe640 10 client.4601.objecter ms_handle_connect 0x7f686806dbd0 2024-03-29T10:43:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.526+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6874050790 con 0x7f687c0b7bb0 2024-03-29T10:43:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.526+0000 7f68637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:21.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.526+0000 7f68637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:21.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.843+0000 7f6881577640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:21.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.843+0000 7f6881577640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:21.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.843+0000 7f6881577640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:21.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.843+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f685c005330 con 0x7f687c0b7bb0 2024-03-29T10:43:21.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.845+0000 7f68637fe640 1 -- 172.21.15.117:0/607513960 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f68740526c0 con 0x7f687c0b7bb0 2024-03-29T10:43:21.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.845+0000 7f68637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:21.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.845+0000 7f68637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:21.831 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:43:21.831 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:21.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.849+0000 7f6881577640 10 librados: watch_flush enter 2024-03-29T10:43:21.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.849+0000 7f6881577640 10 librados: watch_flush exit 2024-03-29T10:43:21.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.849+0000 7f6881577640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T10:43:21.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.849+0000 7f6881577640 10 client.4601.objecter successfully canceled tick 2024-03-29T10:43:21.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.849+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 >> v1:172.21.15.117:6816/376768171 conn(0x7f686806dbd0 legacy=0x7f6868070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:21.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.849+0000 7f6881577640 10 monclient: shutdown 2024-03-29T10:43:21.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.849+0000 7f6881577640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:21.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.849+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 >> v1:172.21.15.117:6790/0 conn(0x7f687c0b7bb0 legacy=0x7f687c1e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:21.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.850+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 shutdown_connections 2024-03-29T10:43:21.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.850+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 >> 172.21.15.117:0/607513960 conn(0x7f687c066a00 msgr2=0x7f687c141550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:21.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.850+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 shutdown_connections 2024-03-29T10:43:21.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.850+0000 7f6881577640 1 -- 172.21.15.117:0/607513960 wait complete. 2024-03-29T10:43:21.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:21.850+0000 7f6881577640 1 librados: shutdown 2024-03-29T10:43:21.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:28.657 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.821+0000 7f0a678c1640 10 monclient: get_monmap_and_config 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.821+0000 7f0a678c1640 10 monclient: build_initial_monmap 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.822+0000 7f0a678c1640 10 monclient: monmap: 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:43:28.822926+0000 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:43:28.822926+0000 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:28.805 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:43:28.806 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:43:28.806 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:28.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.823+0000 7f0a678c1640 1 Processor -- start 2024-03-29T10:43:28.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.823+0000 7f0a678c1640 1 -- start start 2024-03-29T10:43:28.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.823+0000 7f0a678c1640 10 monclient: init 2024-03-29T10:43:28.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:28.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:28.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0a601457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:28.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0a6013f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:28.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0a600a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:28.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a600aaef0 con 0x7f0a600a9d70 2024-03-29T10:43:28.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a60145bd0 con 0x7f0a6013f650 2024-03-29T10:43:28.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a60145eb0 con 0x7f0a601457f0 2024-03-29T10:43:28.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:28.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.824+0000 7f0a678c1640 10 monclient(hunting): authenticate will time out at 1330.086829s 2024-03-29T10:43:28.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.826+0000 7f0a6565d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f0a601457f0 0x7f0a6013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:33198/0 (socket says 172.21.15.117:33198) 2024-03-29T10:43:28.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.826+0000 7f0a6565d640 1 -- 172.21.15.117:0/2814213791 learned_addr learned my addr 172.21.15.117:0/2814213791 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652225713 0 0) 0x7f0a60145bd0 con 0x7f0a6013f650 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 10 monclient(hunting): my global_id is 4510 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a50003520 con 0x7f0a6013f650 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882297392 0 0) 0x7f0a60145eb0 con 0x7f0a601457f0 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:28.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:28.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 10 monclient(hunting): my global_id is 4604 2024-03-29T10:43:28.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.829+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a60145bd0 con 0x7f0a601457f0 2024-03-29T10:43:28.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143630888 0 0) 0x7f0a600aaef0 con 0x7f0a600a9d70 2024-03-29T10:43:28.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:28.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:28.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:28.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient(hunting): my global_id is 4542 2024-03-29T10:43:28.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a60145eb0 con 0x7f0a600a9d70 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683837971 0 0) 0x7f0a60145eb0 con 0x7f0a600a9d70 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 >> v1:172.21.15.117:6790/0 conn(0x7f0a601457f0 legacy=0x7f0a6013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 >> v1:172.21.15.136:6789/0 conn(0x7f0a6013f650 legacy=0x7f0a600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 1 monclient: found mon.noname-a 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60146340 con 0x7f0a600a9d70 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient: _finish_auth 0 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a678c1640 5 monclient: authenticate success, global_id 4542 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a678c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0a58001570 con 0x7f0a600a9d70 2024-03-29T10:43:28.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient: dump: 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a678c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 1 -- 172.21.15.117:0/2814213791 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a58001740 con 0x7f0a600a9d70 2024-03-29T10:43:28.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:28.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a678c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:28.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.830+0000 7f0a678c1640 4 set_mon_vals no callback set 2024-03-29T10:43:28.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.831+0000 7f0a678c1640 10 monclient: shutdown 2024-03-29T10:43:28.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.831+0000 7f0a678c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:28.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.831+0000 7f0a678c1640 1 -- 172.21.15.117:0/2814213791 >> v1:172.21.15.117:6789/0 conn(0x7f0a600a9d70 legacy=0x7f0a6014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:28.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.833+0000 7f0a678c1640 1 -- 172.21.15.117:0/2814213791 shutdown_connections 2024-03-29T10:43:28.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.833+0000 7f0a678c1640 1 -- 172.21.15.117:0/2814213791 >> 172.21.15.117:0/2814213791 conn(0x7f0a60066a00 msgr2=0x7f0a60066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:28.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.833+0000 7f0a678c1640 1 -- 172.21.15.117:0/2814213791 shutdown_connections 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.834+0000 7f0a678c1640 1 -- 172.21.15.117:0/2814213791 wait complete. 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.834+0000 7f0a678c1640 10 monclient: build_initial_monmap 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.834+0000 7f0a678c1640 10 monclient: monmap: 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:28.820 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.835+0000 7f0a678c1640 1 librados: starting msgr at 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.835+0000 7f0a678c1640 1 librados: starting objecter 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.835+0000 7f0a678c1640 1 Processor -- start 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.835+0000 7f0a678c1640 1 -- start start 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.835+0000 7f0a678c1640 1 librados: setting wanted keys 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.835+0000 7f0a678c1640 1 librados: calling monclient init 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.835+0000 7f0a678c1640 10 monclient: init 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:28.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 10 monclient(hunting): picked mon.noname-b con 0x7f0a600a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:28.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 10 monclient(hunting): picked mon.noname-c con 0x7f0a6013f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:28.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 10 monclient(hunting): picked mon.noname-a con 0x7f0a601457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:28.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6006ee40 con 0x7f0a601457f0 2024-03-29T10:43:28.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6013a4e0 con 0x7f0a600a9d70 2024-03-29T10:43:28.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6013aa00 con 0x7f0a6013f650 2024-03-29T10:43:28.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:28.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.836+0000 7f0a678c1640 10 monclient(hunting): authenticate will time out at 1330.098460s 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.838+0000 7f0a64e5c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f0a6013f650 0x7f0a600715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:35826/0 (socket says 172.21.15.117:35826) 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a64e5c640 1 -- 172.21.15.117:0/1590488597 learned_addr learned my addr 172.21.15.117:0/1590488597 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a5dffb640 10 client.?.objecter ms_handle_connect 0x7f0a6013f650 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a5dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866737596 0 0) 0x7f0a6013aa00 con 0x7f0a6013f650 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a5dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a5dffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a44004eb0 con 0x7f0a6013f650 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.839+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3604388101 0 0) 0x7f0a44004eb0 con 0x7f0a6013f650 2024-03-29T10:43:28.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a6013aa00 con 0x7f0a6013f650 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0a54001d50 con 0x7f0a6013f650 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 10 monclient(hunting): dump: 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:28.824 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 10 client.?.objecter ms_handle_connect 0x7f0a600a9d70 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 37740729 0 0) 0x7f0a6013aa00 con 0x7f0a6013f650 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 >> v1:172.21.15.136:6789/0 conn(0x7f0a600a9d70 legacy=0x7f0a601d2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.840+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 >> v1:172.21.15.117:6789/0 conn(0x7f0a601457f0 legacy=0x7f0a601de7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 1 monclient: found mon.c 2024-03-29T10:43:28.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:28.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:28.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6013ab90 con 0x7f0a6013f650 2024-03-29T10:43:28.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:28.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a678c1640 5 monclient: authenticate success, global_id 4607 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a678c1640 10 monclient: _renew_subs 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a678c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a601426b0 con 0x7f0a6013f650 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a678c1640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a678c1640 10 monclient: _renew_subs 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a678c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a601d3780 con 0x7f0a6013f650 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a678c1640 1 librados: init done 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a54001fb0 con 0x7f0a6013f650 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f0a54003de0 con 0x7f0a6013f650 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:28.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5dffb640 10 monclient: dump: 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.842+0000 7f0a5f7fe640 4 set_mon_vals no callback set 2024-03-29T10:43:28.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.843+0000 7f0a678c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:28.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.845+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f0a54003f40 con 0x7f0a6013f650 2024-03-29T10:43:28.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.846+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f0a54084fd0 con 0x7f0a6013f650 2024-03-29T10:43:28.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.846+0000 7f0a5dffb640 10 client.4607.objecter ms_dispatch 0x7f0a60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:28.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.846+0000 7f0a5dffb640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:28.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.846+0000 7f0a5dffb640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:28.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.846+0000 7f0a5dffb640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T10:43:28.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.847+0000 7f0a678c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:28.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.847+0000 7f0a678c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:28.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.847+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a40005080 con 0x7f0a6013f650 2024-03-29T10:43:28.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.847+0000 7f0a6565d640 10 monclient: get_auth_request con 0x7f0a4406dc50 auth_method 0 2024-03-29T10:43:28.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.847+0000 7f0a5dffb640 10 client.4607.objecter ms_handle_connect 0x7f0a4406dc50 2024-03-29T10:43:28.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.855+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a540506e0 con 0x7f0a6013f650 2024-03-29T10:43:28.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.855+0000 7f0a5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:28.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:28.855+0000 7f0a5dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:29.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.180+0000 7f0a678c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:29.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.180+0000 7f0a678c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:29.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.180+0000 7f0a678c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:29.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.180+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a40005330 con 0x7f0a6013f650 2024-03-29T10:43:29.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.181+0000 7f0a5dffb640 1 -- 172.21.15.117:0/1590488597 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f0a540527e0 con 0x7f0a6013f650 2024-03-29T10:43:29.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.181+0000 7f0a5dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:29.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.181+0000 7f0a5dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:29.169 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:43:29.169 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:29.170 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.185+0000 7f0a678c1640 10 librados: watch_flush enter 2024-03-29T10:43:29.170 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.185+0000 7f0a678c1640 10 librados: watch_flush exit 2024-03-29T10:43:29.170 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.185+0000 7f0a678c1640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T10:43:29.170 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.185+0000 7f0a678c1640 10 client.4607.objecter successfully canceled tick 2024-03-29T10:43:29.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.185+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 >> v1:172.21.15.117:6816/376768171 conn(0x7f0a4406dc50 legacy=0x7f0a44070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:29.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.186+0000 7f0a678c1640 10 monclient: shutdown 2024-03-29T10:43:29.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.186+0000 7f0a678c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:29.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.186+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 >> v1:172.21.15.117:6790/0 conn(0x7f0a6013f650 legacy=0x7f0a600715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:29.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.186+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 shutdown_connections 2024-03-29T10:43:29.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.186+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 >> 172.21.15.117:0/1590488597 conn(0x7f0a60066a00 msgr2=0x7f0a6013e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:29.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.186+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 shutdown_connections 2024-03-29T10:43:29.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.186+0000 7f0a678c1640 1 -- 172.21.15.117:0/1590488597 wait complete. 2024-03-29T10:43:29.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:29.186+0000 7f0a678c1640 1 librados: shutdown 2024-03-29T10:43:29.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:35.994 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:36.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.175+0000 7f939a9f6640 10 monclient: get_monmap_and_config 2024-03-29T10:43:36.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.175+0000 7f939a9f6640 10 monclient: build_initial_monmap 2024-03-29T10:43:36.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.175+0000 7f939a9f6640 10 monclient: monmap: 2024-03-29T10:43:36.160 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:36.160 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:36.160 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:43:36.177784+0000 2024-03-29T10:43:36.160 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:43:36.177784+0000 2024-03-29T10:43:36.160 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:36.161 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:36.161 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:36.161 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:43:36.161 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:43:36.161 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:36.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.176+0000 7f939a9f6640 1 Processor -- start 2024-03-29T10:43:36.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.177+0000 7f939a9f6640 1 -- start start 2024-03-29T10:43:36.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.177+0000 7f939a9f6640 10 monclient: init 2024-03-29T10:43:36.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:36.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f93941422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f9394142690 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f93940b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9394142c80 con 0x7f93940b17d0 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93940aae40 con 0x7f93941422b0 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f9398f93640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f93940b17d0 0x7f93940b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:45442/0 (socket says 172.21.15.117:45442) 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f9398f93640 1 -- 172.21.15.117:0/684520613 learned_addr learned my addr 172.21.15.117:0/684520613 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 1 -- 172.21.15.117:0/684520613 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939413fbd0 con 0x7f9394142690 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:36.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.178+0000 7f939a9f6640 10 monclient(hunting): authenticate will time out at 1337.441411s 2024-03-29T10:43:36.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412330455 0 0) 0x7f9394142c80 con 0x7f93940b17d0 2024-03-29T10:43:36.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:36.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:36.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:36.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): my global_id is 4545 2024-03-29T10:43:36.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f937c003520 con 0x7f93940b17d0 2024-03-29T10:43:36.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789175059 0 0) 0x7f93940aae40 con 0x7f93941422b0 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): my global_id is 4513 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9394142c80 con 0x7f93941422b0 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454979695 0 0) 0x7f939413fbd0 con 0x7f9394142690 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.179+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93940aae40 con 0x7f9394142690 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045757705 0 0) 0x7f937c003520 con 0x7f93940b17d0 2024-03-29T10:43:36.168 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 >> v1:172.21.15.117:6790/0 conn(0x7f9394142690 legacy=0x7f93940b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 >> v1:172.21.15.136:6789/0 conn(0x7f93941422b0 legacy=0x7f939413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 1 monclient: found mon.noname-a 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394151940 con 0x7f93940b17d0 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:36.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f938c001a10 con 0x7f93940b17d0 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: dump: 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:36.170 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f939a9f6640 5 monclient: authenticate success, global_id 4545 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f939a9f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f938c001b70 con 0x7f93940b17d0 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 1 -- 172.21.15.117:0/684520613 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f938c001f00 con 0x7f93940b17d0 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f9392ffd640 10 monclient: dump: 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:36.171 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f939a9f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.180+0000 7f939a9f6640 4 set_mon_vals no callback set 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.181+0000 7f939a9f6640 10 monclient: shutdown 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.181+0000 7f939a9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.181+0000 7f939a9f6640 1 -- 172.21.15.117:0/684520613 >> v1:172.21.15.117:6789/0 conn(0x7f93940b17d0 legacy=0x7f93940b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.182+0000 7f939a9f6640 1 -- 172.21.15.117:0/684520613 shutdown_connections 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.182+0000 7f939a9f6640 1 -- 172.21.15.117:0/684520613 >> 172.21.15.117:0/684520613 conn(0x7f9394066a00 msgr2=0x7f9394066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.182+0000 7f939a9f6640 1 -- 172.21.15.117:0/684520613 shutdown_connections 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.182+0000 7f939a9f6640 1 -- 172.21.15.117:0/684520613 wait complete. 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.183+0000 7f939a9f6640 10 monclient: build_initial_monmap 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.183+0000 7f939a9f6640 10 monclient: monmap: 2024-03-29T10:43:36.172 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:36.173 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:36.173 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:43:36.173 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:43:36.173 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:36.173 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:36.173 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:36.173 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:43:36.173 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:43:36.173 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.184+0000 7f939a9f6640 1 librados: starting msgr at 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.184+0000 7f939a9f6640 1 librados: starting objecter 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.184+0000 7f939a9f6640 1 Processor -- start 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.184+0000 7f939a9f6640 1 -- start start 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.184+0000 7f939a9f6640 1 librados: setting wanted keys 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.184+0000 7f939a9f6640 1 librados: calling monclient init 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.184+0000 7f939a9f6640 10 monclient: init 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.185+0000 7f939a9f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.185+0000 7f939a9f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f939a9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f93940b17d0 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:36.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f939a9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f93941422b0 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f939a9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f9394142690 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f9393fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f93940b17d0 0x7f939406de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:43688/0 (socket says 172.21.15.117:43688) 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f9393fff640 1 -- 172.21.15.117:0/1099806827 learned_addr learned my addr 172.21.15.117:0/1099806827 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939414e750 con 0x7f93941422b0 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f9390ff9640 10 client.?.objecter ms_handle_connect 0x7f93940b17d0 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f9390ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939414e4b0 con 0x7f93940b17d0 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939406fad0 con 0x7f9394142690 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f939a9f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f939a9f6640 10 monclient(hunting): authenticate will time out at 1337.449671s 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f9390ff9640 10 client.?.objecter ms_handle_connect 0x7f93941422b0 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.186+0000 7f9390ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:36.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129871166 0 0) 0x7f939414e4b0 con 0x7f93940b17d0 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9380004eb0 con 0x7f93940b17d0 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 client.?.objecter ms_handle_connect 0x7f9394142690 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074353703 0 0) 0x7f939414e750 con 0x7f93941422b0 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:36.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f939414e4b0 con 0x7f93941422b0 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 560131891 0 0) 0x7f9380004eb0 con 0x7f93940b17d0 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f939414e750 con 0x7f93940b17d0 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f9384001a30 con 0x7f93940b17d0 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): dump: 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:36.183 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328983839 0 0) 0x7f939406fad0 con 0x7f9394142690 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:36.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 10 monclient(hunting): my global_id is 4613 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.187+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9380004eb0 con 0x7f9394142690 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1782207488 0 0) 0x7f939414e750 con 0x7f93940b17d0 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 >> v1:172.21.15.117:6790/0 conn(0x7f9394142690 legacy=0x7f93941e6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 >> v1:172.21.15.117:6789/0 conn(0x7f93941422b0 legacy=0x7f939413bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 1 monclient: found mon.b 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939406fdb0 con 0x7f93940b17d0 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 10 monclient: _finish_auth 0 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f9390ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f939a9f6640 5 monclient: authenticate success, global_id 4516 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f939a9f6640 10 monclient: _renew_subs 2024-03-29T10:43:36.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f939a9f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.188+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9394070090 con 0x7f93940b17d0 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f939a9f6640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f939a9f6640 10 monclient: _renew_subs 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f939a9f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f939406e5a0 con 0x7f93940b17d0 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f939a9f6640 1 librados: init done 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9384001b90 con 0x7f93940b17d0 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f9390ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f93840038c0 con 0x7f93940b17d0 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f9390ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f9390ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f9390ff9640 10 monclient: dump: 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:36.186 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.189+0000 7f9391ffb640 4 set_mon_vals no callback set 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.190+0000 7f939a9f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.191+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f9384003a20 con 0x7f93940b17d0 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.191+0000 7f939a9f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:36.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.191+0000 7f939a9f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.191+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9374005080 con 0x7f93940b17d0 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.192+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f9384084b00 con 0x7f93940b17d0 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.192+0000 7f9390ff9640 10 client.4516.objecter ms_dispatch 0x7f9394000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.192+0000 7f9390ff9640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.192+0000 7f9390ff9640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.192+0000 7f9390ff9640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.192+0000 7f93937fe640 10 monclient: get_auth_request con 0x7f938006dfa0 auth_method 0 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.193+0000 7f9390ff9640 10 client.4516.objecter ms_handle_connect 0x7f938006dfa0 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.199+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9384050400 con 0x7f93940b17d0 2024-03-29T10:43:36.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.199+0000 7f9390ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:36.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.199+0000 7f9390ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:36.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.615+0000 7f939a9f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:36.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.615+0000 7f939a9f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:36.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.615+0000 7f939a9f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:43:36.599 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.615+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9374005330 con 0x7f93940b17d0 2024-03-29T10:43:36.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.616+0000 7f9390ff9640 1 -- 172.21.15.117:0/1099806827 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f9384052330 con 0x7f93940b17d0 2024-03-29T10:43:36.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.616+0000 7f9390ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:36.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.616+0000 7f9390ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:36.600 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:43:36.601 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:36.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 10 librados: watch_flush enter 2024-03-29T10:43:36.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 10 librados: watch_flush exit 2024-03-29T10:43:36.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T10:43:36.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 10 client.4516.objecter successfully canceled tick 2024-03-29T10:43:36.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 >> v1:172.21.15.117:6816/376768171 conn(0x7f938006dfa0 legacy=0x7f9380070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:36.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 10 monclient: shutdown 2024-03-29T10:43:36.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:36.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 >> v1:172.21.15.136:6789/0 conn(0x7f93940b17d0 legacy=0x7f939406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:36.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 shutdown_connections 2024-03-29T10:43:36.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.620+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 >> 172.21.15.117:0/1099806827 conn(0x7f9394066a00 msgr2=0x7f939413d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:36.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.621+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 shutdown_connections 2024-03-29T10:43:36.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.621+0000 7f939a9f6640 1 -- 172.21.15.117:0/1099806827 wait complete. 2024-03-29T10:43:36.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:36.621+0000 7f939a9f6640 1 librados: shutdown 2024-03-29T10:43:36.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:43.427 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.584+0000 7f7b6138d640 10 monclient: get_monmap_and_config 2024-03-29T10:43:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.584+0000 7f7b6138d640 10 monclient: build_initial_monmap 2024-03-29T10:43:43.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.584+0000 7f7b6138d640 10 monclient: monmap: 2024-03-29T10:43:43.568 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:43.568 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:43.568 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:43:43.585834+0000 2024-03-29T10:43:43.568 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:43:43.585834+0000 2024-03-29T10:43:43.568 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:43.568 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:43.569 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:43.569 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:43:43.569 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:43:43.569 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.585+0000 7f7b6138d640 1 Processor -- start 2024-03-29T10:43:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.586+0000 7f7b6138d640 1 -- start start 2024-03-29T10:43:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.586+0000 7f7b6138d640 10 monclient: init 2024-03-29T10:43:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.587+0000 7f7b6138d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.587+0000 7f7b6138d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.587+0000 7f7b6138d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5c0b4890 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:43.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.587+0000 7f7b6138d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b5c13c040 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.587+0000 7f7b5a7fc640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f7b5c13c040 0x7f7b5c1420d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58748/0 (socket says 172.21.15.117:58748) 2024-03-29T10:43:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b6138d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b5c1427e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b6138d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c0b42f0 con 0x7f7b5c0b4890 2024-03-29T10:43:43.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b6138d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c0aa7a0 con 0x7f7b5c1427e0 2024-03-29T10:43:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b6138d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c0aaa10 con 0x7f7b5c13c040 2024-03-29T10:43:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b6138d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:43.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b6138d640 10 monclient(hunting): authenticate will time out at 1344.850425s 2024-03-29T10:43:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b5a7fc640 1 -- 172.21.15.117:0/410339474 learned_addr learned my addr 172.21.15.117:0/410339474 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b59ffb640 1 -- 172.21.15.117:0/410339474 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681675720 0 0) 0x7f7b5c0aaa10 con 0x7f7b5c13c040 2024-03-29T10:43:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b59ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b59ffb640 10 monclient(hunting): my global_id is 4616 2024-03-29T10:43:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.588+0000 7f7b59ffb640 1 -- 172.21.15.117:0/410339474 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b44003520 con 0x7f7b5c13c040 2024-03-29T10:43:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.589+0000 7f7b59ffb640 1 -- 172.21.15.117:0/410339474 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2184565699 0 0) 0x7f7b44003520 con 0x7f7b5c13c040 2024-03-29T10:43:43.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.589+0000 7f7b59ffb640 1 -- 172.21.15.117:0/410339474 >> v1:172.21.15.136:6789/0 conn(0x7f7b5c1427e0 legacy=0x7f7b5c14d880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:43.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 1 -- 172.21.15.117:0/410339474 >> v1:172.21.15.117:6789/0 conn(0x7f7b5c0b4890 legacy=0x7f7b5c13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 1 monclient: found mon.noname-b 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 1 -- 172.21.15.117:0/410339474 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c0aacf0 con 0x7f7b5c13c040 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 1 -- 172.21.15.117:0/410339474 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7b50001670 con 0x7f7b5c13c040 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b59ffb640 10 monclient: dump: 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b6138d640 5 monclient: authenticate success, global_id 4616 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.591+0000 7f7b6138d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.592+0000 7f7b59ffb640 1 -- 172.21.15.117:0/410339474 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b500018d0 con 0x7f7b5c13c040 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.592+0000 7f7b59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.592+0000 7f7b6138d640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:43.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.592+0000 7f7b6138d640 4 set_mon_vals no callback set 2024-03-29T10:43:43.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.593+0000 7f7b6138d640 10 monclient: shutdown 2024-03-29T10:43:43.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.593+0000 7f7b6138d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:43.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.593+0000 7f7b6138d640 1 -- 172.21.15.117:0/410339474 >> v1:172.21.15.117:6790/0 conn(0x7f7b5c13c040 legacy=0x7f7b5c1420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.593+0000 7f7b59ffb640 1 -- 172.21.15.117:0/410339474 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7b50001ca0 con 0x7f7b5c13c040 2024-03-29T10:43:43.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.593+0000 7f7b59ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:43:43.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.593+0000 7f7b6138d640 1 -- 172.21.15.117:0/410339474 shutdown_connections 2024-03-29T10:43:43.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.593+0000 7f7b6138d640 1 -- 172.21.15.117:0/410339474 >> 172.21.15.117:0/410339474 conn(0x7f7b5c06d180 msgr2=0x7f7b5c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:43.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.593+0000 7f7b6138d640 1 -- 172.21.15.117:0/410339474 shutdown_connections 2024-03-29T10:43:43.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.593+0000 7f7b6138d640 1 -- 172.21.15.117:0/410339474 wait complete. 2024-03-29T10:43:43.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.595+0000 7f7b6138d640 10 monclient: build_initial_monmap 2024-03-29T10:43:43.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.595+0000 7f7b6138d640 10 monclient: monmap: 2024-03-29T10:43:43.581 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:43.581 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.595+0000 7f7b6138d640 1 librados: starting msgr at 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.595+0000 7f7b6138d640 1 librados: starting objecter 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.596+0000 7f7b6138d640 1 Processor -- start 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.596+0000 7f7b6138d640 1 -- start start 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.596+0000 7f7b6138d640 1 librados: setting wanted keys 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.596+0000 7f7b6138d640 1 librados: calling monclient init 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.596+0000 7f7b6138d640 10 monclient: init 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b5c0b4890 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b5c13c040 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5c1427e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c0add00 con 0x7f7b5c1427e0 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c1d6000 con 0x7f7b5c0b4890 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c1d62a0 con 0x7f7b5c13c040 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.597+0000 7f7b6138d640 10 monclient(hunting): authenticate will time out at 1344.859992s 2024-03-29T10:43:43.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.600+0000 7f7b5affd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7b5c0b4890 0x7f7b5c1d67c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:55513/0 (socket says 172.21.15.117:55513) 2024-03-29T10:43:43.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.600+0000 7f7b5affd640 1 -- 172.21.15.117:0/1943595625 learned_addr learned my addr 172.21.15.117:0/1943595625 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:43.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.601+0000 7f7b437fe640 10 client.?.objecter ms_handle_connect 0x7f7b5c13c040 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.601+0000 7f7b437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.601+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877687208 0 0) 0x7f7b5c1d62a0 con 0x7f7b5c13c040 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.601+0000 7f7b437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.601+0000 7f7b437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.601+0000 7f7b437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.601+0000 7f7b437fe640 10 monclient(hunting): my global_id is 4619 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.601+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b48004eb0 con 0x7f7b5c13c040 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.601+0000 7f7b437fe640 10 client.?.objecter ms_handle_connect 0x7f7b5c0b4890 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116806326 0 0) 0x7f7b5c1d6000 con 0x7f7b5c0b4890 2024-03-29T10:43:43.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): my global_id is 4519 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c1d62a0 con 0x7f7b5c0b4890 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 client.?.objecter ms_handle_connect 0x7f7b5c1427e0 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 816503113 0 0) 0x7f7b48004eb0 con 0x7f7b5c13c040 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b5c1d6000 con 0x7f7b5c13c040 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344087976 0 0) 0x7f7b5c0add00 con 0x7f7b5c1427e0 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): my global_id is 4551 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b48004eb0 con 0x7f7b5c1427e0 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7b50001be0 con 0x7f7b5c13c040 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): dump: 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:43.588 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:43.589 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:43.589 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:43.589 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:43.589 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:43.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:43.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 265091698 0 0) 0x7f7b48004eb0 con 0x7f7b5c1427e0 2024-03-29T10:43:43.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.602+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b5c0add00 con 0x7f7b5c1427e0 2024-03-29T10:43:43.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1757077838 0 0) 0x7f7b5c1d6000 con 0x7f7b5c13c040 2024-03-29T10:43:43.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 >> v1:172.21.15.136:6789/0 conn(0x7f7b5c0b4890 legacy=0x7f7b5c1d67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 >> v1:172.21.15.117:6789/0 conn(0x7f7b5c1427e0 legacy=0x7f7b5c1de380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:43.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 1 monclient: found mon.c 2024-03-29T10:43:43.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:43.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:43.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c1d6ed0 con 0x7f7b5c13c040 2024-03-29T10:43:43.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:43.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:43.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b6138d640 5 monclient: authenticate success, global_id 4619 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b6138d640 10 monclient: _renew_subs 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b6138d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b6138d640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5c0acc00 con 0x7f7b5c13c040 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b6138d640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b6138d640 10 monclient: _renew_subs 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b6138d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b6138d640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5c0abca0 con 0x7f7b5c13c040 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.603+0000 7f7b6138d640 1 librados: init done 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b50004bf0 con 0x7f7b5c13c040 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:43.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7b50001d40 con 0x7f7b5c13c040 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b437fe640 10 monclient: dump: 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:43.594 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:43.595 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:43.595 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:43.595 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:43.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b417fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:43.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b417fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:43.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b417fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:43.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.605+0000 7f7b417fa640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3c005080 con 0x7f7b5c13c040 2024-03-29T10:43:43.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.607+0000 7f7b58ff9640 4 set_mon_vals no callback set 2024-03-29T10:43:43.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.609+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f7b5001ad70 con 0x7f7b5c13c040 2024-03-29T10:43:43.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.610+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f7b50084b70 con 0x7f7b5c13c040 2024-03-29T10:43:43.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.610+0000 7f7b437fe640 10 client.4619.objecter ms_dispatch 0x7f7b5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:43.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.610+0000 7f7b437fe640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:43.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.610+0000 7f7b437fe640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:43.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.610+0000 7f7b437fe640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T10:43:43.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.610+0000 7f7b5affd640 10 monclient: get_auth_request con 0x7f7b4806df00 auth_method 0 2024-03-29T10:43:43.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.611+0000 7f7b437fe640 10 client.4619.objecter ms_handle_connect 0x7f7b4806df00 2024-03-29T10:43:43.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.615+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b50050450 con 0x7f7b5c13c040 2024-03-29T10:43:43.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.615+0000 7f7b437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:43.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.615+0000 7f7b437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:43.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.951+0000 7f7b417fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:43.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.951+0000 7f7b417fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:43.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.951+0000 7f7b417fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:43.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.951+0000 7f7b417fa640 1 -- 172.21.15.117:0/1943595625 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b3c005330 con 0x7f7b5c13c040 2024-03-29T10:43:43.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.952+0000 7f7b437fe640 1 -- 172.21.15.117:0/1943595625 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f7b50052380 con 0x7f7b5c13c040 2024-03-29T10:43:43.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.952+0000 7f7b437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:43.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.952+0000 7f7b437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:43.936 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:43:43.936 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:43.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 10 librados: watch_flush enter 2024-03-29T10:43:43.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 10 librados: watch_flush exit 2024-03-29T10:43:43.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T10:43:43.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 10 client.4619.objecter successfully canceled tick 2024-03-29T10:43:43.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 1 -- 172.21.15.117:0/1943595625 >> v1:172.21.15.117:6816/376768171 conn(0x7f7b4806df00 legacy=0x7f7b480703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 10 monclient: shutdown 2024-03-29T10:43:43.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:43.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 1 -- 172.21.15.117:0/1943595625 >> v1:172.21.15.117:6790/0 conn(0x7f7b5c13c040 legacy=0x7f7b5c139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 1 -- 172.21.15.117:0/1943595625 shutdown_connections 2024-03-29T10:43:43.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 1 -- 172.21.15.117:0/1943595625 >> 172.21.15.117:0/1943595625 conn(0x7f7b5c06d180 msgr2=0x7f7b5c0b8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:43.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 1 -- 172.21.15.117:0/1943595625 shutdown_connections 2024-03-29T10:43:43.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 1 -- 172.21.15.117:0/1943595625 wait complete. 2024-03-29T10:43:43.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:43.956+0000 7f7b417fa640 1 librados: shutdown 2024-03-29T10:43:43.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:50.762 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:50.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.934+0000 7fbf54893640 10 monclient: get_monmap_and_config 2024-03-29T10:43:50.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.934+0000 7fbf54893640 10 monclient: build_initial_monmap 2024-03-29T10:43:50.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.934+0000 7fbf54893640 10 monclient: monmap: 2024-03-29T10:43:50.918 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:50.919 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:50.919 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:43:50.935780+0000 2024-03-29T10:43:50.919 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:43:50.935780+0000 2024-03-29T10:43:50.919 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:50.919 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:50.919 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:50.919 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:43:50.919 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:43:50.919 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:50.920 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.937+0000 7fbf54893640 1 Processor -- start 2024-03-29T10:43:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.939+0000 7fbf54893640 1 -- start start 2024-03-29T10:43:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.939+0000 7fbf54893640 10 monclient: init 2024-03-29T10:43:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:50.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 10 monclient(hunting): picked mon.noname-b con 0x7fbf501457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5013f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 10 monclient(hunting): picked mon.noname-a con 0x7fbf500a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf500aaef0 con 0x7fbf500a9d70 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf50145bd0 con 0x7fbf5013f650 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf50145eb0 con 0x7fbf501457f0 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf54893640 10 monclient(hunting): authenticate will time out at 1352.202726s 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf4e59c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fbf501457f0 0x7fbf5013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:38414/0 (socket says 172.21.15.117:38414) 2024-03-29T10:43:50.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.940+0000 7fbf4e59c640 1 -- 172.21.15.117:0/3987813158 learned_addr learned my addr 172.21.15.117:0/3987813158 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:50.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027452674 0 0) 0x7fbf50145eb0 con 0x7fbf501457f0 2024-03-29T10:43:50.931 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf38003520 con 0x7fbf501457f0 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786925233 0 0) 0x7fbf500aaef0 con 0x7fbf500a9d70 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient(hunting): my global_id is 4554 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf50145eb0 con 0x7fbf500a9d70 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83241385 0 0) 0x7fbf38003520 con 0x7fbf501457f0 2024-03-29T10:43:50.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 >> v1:172.21.15.136:6789/0 conn(0x7fbf5013f650 legacy=0x7fbf500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 >> v1:172.21.15.117:6789/0 conn(0x7fbf500a9d70 legacy=0x7fbf5014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 monclient: found mon.noname-b 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50146340 con 0x7fbf501457f0 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient: _finish_auth 0 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf54893640 5 monclient: authenticate success, global_id 4622 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf54893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:50.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbf44001690 con 0x7fbf501457f0 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient: dump: 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:50.934 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf54893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 1 -- 172.21.15.117:0/3987813158 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf440018b0 con 0x7fbf501457f0 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf4d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf54893640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.945+0000 7fbf54893640 4 set_mon_vals no callback set 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.946+0000 7fbf54893640 10 monclient: shutdown 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.946+0000 7fbf54893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.946+0000 7fbf54893640 1 -- 172.21.15.117:0/3987813158 >> v1:172.21.15.117:6790/0 conn(0x7fbf501457f0 legacy=0x7fbf5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:50.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.948+0000 7fbf54893640 1 -- 172.21.15.117:0/3987813158 shutdown_connections 2024-03-29T10:43:50.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.948+0000 7fbf54893640 1 -- 172.21.15.117:0/3987813158 >> 172.21.15.117:0/3987813158 conn(0x7fbf50066a00 msgr2=0x7fbf50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:50.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.948+0000 7fbf54893640 1 -- 172.21.15.117:0/3987813158 shutdown_connections 2024-03-29T10:43:50.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.948+0000 7fbf54893640 1 -- 172.21.15.117:0/3987813158 wait complete. 2024-03-29T10:43:50.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.949+0000 7fbf54893640 10 monclient: build_initial_monmap 2024-03-29T10:43:50.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.949+0000 7fbf54893640 10 monclient: monmap: 2024-03-29T10:43:50.944 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:50.944 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:50.944 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.949+0000 7fbf54893640 1 librados: starting msgr at 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.949+0000 7fbf54893640 1 librados: starting objecter 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.950+0000 7fbf54893640 1 Processor -- start 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.950+0000 7fbf54893640 1 -- start start 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.950+0000 7fbf54893640 1 librados: setting wanted keys 2024-03-29T10:43:50.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.950+0000 7fbf54893640 1 librados: calling monclient init 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.950+0000 7fbf54893640 10 monclient: init 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.950+0000 7fbf54893640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.950+0000 7fbf54893640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.950+0000 7fbf54893640 10 monclient(hunting): picked mon.noname-a con 0x7fbf500a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.950+0000 7fbf54893640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5013f650 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.951+0000 7fbf54893640 10 monclient(hunting): picked mon.noname-c con 0x7fbf501457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.951+0000 7fbf54893640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf501429a0 con 0x7fbf500a9d70 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.951+0000 7fbf54893640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf50141be0 con 0x7fbf5013f650 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.951+0000 7fbf54893640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf50142580 con 0x7fbf501457f0 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.951+0000 7fbf54893640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.951+0000 7fbf54893640 10 monclient(hunting): authenticate will time out at 1352.213147s 2024-03-29T10:43:50.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf4ed9d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fbf501457f0 0x7fbf501d2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41432/0 (socket says 172.21.15.117:41432) 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf4ed9d640 1 -- 172.21.15.117:0/79283124 learned_addr learned my addr 172.21.15.117:0/79283124 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf3effd640 10 client.?.objecter ms_handle_connect 0x7fbf501457f0 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038359109 0 0) 0x7fbf50142580 con 0x7fbf501457f0 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf3effd640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.954+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf30004eb0 con 0x7fbf501457f0 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 client.?.objecter ms_handle_connect 0x7fbf5013f650 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:50.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169372359 0 0) 0x7fbf50141be0 con 0x7fbf5013f650 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 monclient(hunting): my global_id is 4522 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf50142580 con 0x7fbf5013f650 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1329449163 0 0) 0x7fbf30004eb0 con 0x7fbf501457f0 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf50141be0 con 0x7fbf501457f0 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbf40001990 con 0x7fbf501457f0 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 monclient(hunting): dump: 2024-03-29T10:43:50.948 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.955+0000 7fbf3effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2758596419 0 0) 0x7fbf50142580 con 0x7fbf5013f650 2024-03-29T10:43:50.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf30004eb0 con 0x7fbf5013f650 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbf48001bb0 con 0x7fbf5013f650 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient(hunting): dump: 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:50.950 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 client.?.objecter ms_handle_connect 0x7fbf500a9d70 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577651814 0 0) 0x7fbf501429a0 con 0x7fbf500a9d70 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient(hunting): my global_id is 4557 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf50142580 con 0x7fbf500a9d70 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3458905686 0 0) 0x7fbf50141be0 con 0x7fbf501457f0 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 >> v1:172.21.15.136:6789/0 conn(0x7fbf5013f650 legacy=0x7fbf50136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:50.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 >> v1:172.21.15.117:6789/0 conn(0x7fbf500a9d70 legacy=0x7fbf50141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 monclient: found mon.c 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf501445e0 con 0x7fbf501457f0 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient: _finish_auth 0 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf54893640 5 monclient: authenticate success, global_id 4625 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf54893640 10 monclient: _renew_subs 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf54893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:50.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf50144850 con 0x7fbf501457f0 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf54893640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf54893640 10 monclient: _renew_subs 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf54893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf501375b0 con 0x7fbf501457f0 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf54893640 1 librados: init done 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf40001af0 con 0x7fbf501457f0 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.956+0000 7fbf3ffff640 4 set_mon_vals no callback set 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.957+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fbf40003a60 con 0x7fbf501457f0 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.957+0000 7fbf3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.957+0000 7fbf3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.957+0000 7fbf3effd640 10 monclient: dump: 2024-03-29T10:43:50.953 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.959+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fbf40003bc0 con 0x7fbf501457f0 2024-03-29T10:43:50.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.960+0000 7fbf4e59c640 10 monclient: get_auth_request con 0x7fbf3006dfe0 auth_method 0 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.960+0000 7fbf54893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.960+0000 7fbf54893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.960+0000 7fbf54893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.960+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf34005080 con 0x7fbf501457f0 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.960+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fbf40084ab0 con 0x7fbf501457f0 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.960+0000 7fbf3effd640 10 client.4625.objecter ms_dispatch 0x7fbf50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.960+0000 7fbf3effd640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.960+0000 7fbf3effd640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.961+0000 7fbf3effd640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-29T10:43:50.955 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.961+0000 7fbf3effd640 10 client.4625.objecter ms_handle_connect 0x7fbf3006dfe0 2024-03-29T10:43:50.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.973+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf400501c0 con 0x7fbf501457f0 2024-03-29T10:43:50.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.973+0000 7fbf3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:50.956 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:50.973+0000 7fbf3effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:51.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.303+0000 7fbf54893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:51.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.303+0000 7fbf54893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:51.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.303+0000 7fbf54893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:51.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.303+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf34005330 con 0x7fbf501457f0 2024-03-29T10:43:51.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.305+0000 7fbf3effd640 1 -- 172.21.15.117:0/79283124 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fbf400522c0 con 0x7fbf501457f0 2024-03-29T10:43:51.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.305+0000 7fbf3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:51.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.305+0000 7fbf3effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:51.289 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:43:51.289 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:51.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.308+0000 7fbf54893640 10 librados: watch_flush enter 2024-03-29T10:43:51.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.309+0000 7fbf54893640 10 librados: watch_flush exit 2024-03-29T10:43:51.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.309+0000 7fbf54893640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-29T10:43:51.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.309+0000 7fbf54893640 10 client.4625.objecter successfully canceled tick 2024-03-29T10:43:51.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.310+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 >> v1:172.21.15.117:6816/376768171 conn(0x7fbf3006dfe0 legacy=0x7fbf300704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:51.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.310+0000 7fbf54893640 10 monclient: shutdown 2024-03-29T10:43:51.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.310+0000 7fbf54893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:51.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.310+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 >> v1:172.21.15.117:6790/0 conn(0x7fbf501457f0 legacy=0x7fbf501d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:51.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.310+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 shutdown_connections 2024-03-29T10:43:51.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.310+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 >> 172.21.15.117:0/79283124 conn(0x7fbf50066a00 msgr2=0x7fbf500b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:51.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.311+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 shutdown_connections 2024-03-29T10:43:51.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.311+0000 7fbf54893640 1 -- 172.21.15.117:0/79283124 wait complete. 2024-03-29T10:43:51.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:51.311+0000 7fbf54893640 1 librados: shutdown 2024-03-29T10:43:51.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:58.118 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.268+0000 7fd23297c640 10 monclient: get_monmap_and_config 2024-03-29T10:43:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.268+0000 7fd23297c640 10 monclient: build_initial_monmap 2024-03-29T10:43:58.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.268+0000 7fd23297c640 10 monclient: monmap: 2024-03-29T10:43:58.251 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:58.252 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:58.252 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:43:58.269375+0000 2024-03-29T10:43:58.252 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:43:58.269375+0000 2024-03-29T10:43:58.252 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:58.252 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:58.252 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:58.252 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:43:58.252 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:43:58.252 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:58.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.269+0000 7fd23297c640 1 Processor -- start 2024-03-29T10:43:58.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.269+0000 7fd23297c640 1 -- start start 2024-03-29T10:43:58.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.269+0000 7fd23297c640 10 monclient: init 2024-03-29T10:43:58.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.270+0000 7fd23297c640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:58.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.270+0000 7fd23297c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:58.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd23297c640 10 monclient(hunting): picked mon.noname-a con 0x7fd22c1457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:58.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd23297c640 10 monclient(hunting): picked mon.noname-b con 0x7fd22c13f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:58.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd23297c640 10 monclient(hunting): picked mon.noname-c con 0x7fd22c0a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:58.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd23297c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c0aaef0 con 0x7fd22c1457f0 2024-03-29T10:43:58.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd23297c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c145bd0 con 0x7fd22c0a9d70 2024-03-29T10:43:58.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd23297c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c145eb0 con 0x7fd22c13f650 2024-03-29T10:43:58.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd23297c640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:58.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd23297c640 10 monclient(hunting): authenticate will time out at 1359.534314s 2024-03-29T10:43:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd22b7fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd22c13f650 0x7fd22c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:48628/0 (socket says 172.21.15.117:48628) 2024-03-29T10:43:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.272+0000 7fd22b7fe640 1 -- 172.21.15.117:0/3041468577 learned_addr learned my addr 172.21.15.117:0/3041468577 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 -- 172.21.15.117:0/3041468577 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914159532 0 0) 0x7fd22c145eb0 con 0x7fd22c13f650 2024-03-29T10:43:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:43:58.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 -- 172.21.15.117:0/3041468577 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd214003520 con 0x7fd22c13f650 2024-03-29T10:43:58.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 -- 172.21.15.117:0/3041468577 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2750149283 0 0) 0x7fd214003520 con 0x7fd22c13f650 2024-03-29T10:43:58.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 -- 172.21.15.117:0/3041468577 >> v1:172.21.15.136:6789/0 conn(0x7fd22c0a9d70 legacy=0x7fd22c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 -- 172.21.15.117:0/3041468577 >> v1:172.21.15.117:6789/0 conn(0x7fd22c1457f0 legacy=0x7fd22c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 monclient: found mon.noname-b 2024-03-29T10:43:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:58.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 -- 172.21.15.117:0/3041468577 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c146340 con 0x7fd22c13f650 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: _finish_auth 0 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 -- 172.21.15.117:0/3041468577 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd220001670 con 0x7fd22c13f650 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: dump: 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd23297c640 5 monclient: authenticate success, global_id 4628 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd23297c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 -- 172.21.15.117:0/3041468577 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd220001900 con 0x7fd22c13f650 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd23297c640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd23297c640 4 set_mon_vals no callback set 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 1 -- 172.21.15.117:0/3041468577 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd220001670 con 0x7fd22c13f650 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.273+0000 7fd22affd640 10 monclient: dump: 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.268 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.274+0000 7fd23297c640 10 monclient: shutdown 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.274+0000 7fd23297c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.274+0000 7fd23297c640 1 -- 172.21.15.117:0/3041468577 >> v1:172.21.15.117:6790/0 conn(0x7fd22c13f650 legacy=0x7fd22c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.274+0000 7fd23297c640 1 -- 172.21.15.117:0/3041468577 shutdown_connections 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.274+0000 7fd23297c640 1 -- 172.21.15.117:0/3041468577 >> 172.21.15.117:0/3041468577 conn(0x7fd22c066a00 msgr2=0x7fd22c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:58.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.274+0000 7fd23297c640 1 -- 172.21.15.117:0/3041468577 shutdown_connections 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.274+0000 7fd23297c640 1 -- 172.21.15.117:0/3041468577 wait complete. 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.276+0000 7fd23297c640 10 monclient: build_initial_monmap 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.276+0000 7fd23297c640 10 monclient: monmap: 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:43:58.270 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.276+0000 7fd23297c640 1 librados: starting msgr at 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.276+0000 7fd23297c640 1 librados: starting objecter 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.276+0000 7fd23297c640 1 Processor -- start 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 1 -- start start 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 1 librados: setting wanted keys 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 1 librados: calling monclient init 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 10 monclient: init 2024-03-29T10:43:58.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 10 monclient(hunting): picked mon.noname-b con 0x7fd22c0a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 10 monclient(hunting): picked mon.noname-a con 0x7fd22c13f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 10 monclient(hunting): picked mon.noname-c con 0x7fd22c1457f0 addr v1:172.21.15.117:6790/0 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c071890 con 0x7fd22c13f650 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c13c000 con 0x7fd22c0a9d70 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c0ac400 con 0x7fd22c1457f0 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.277+0000 7fd23297c640 10 monclient(hunting): authenticate will time out at 1359.539908s 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd22bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd22c0a9d70 0x7fd22c070a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:43694/0 (socket says 172.21.15.117:43694) 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd22bfff640 1 -- 172.21.15.117:0/1725155785 learned_addr learned my addr 172.21.15.117:0/1725155785 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd228ff9640 10 client.?.objecter ms_handle_connect 0x7fd22c0a9d70 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd228ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594625922 0 0) 0x7fd22c13c000 con 0x7fd22c0a9d70 2024-03-29T10:43:58.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd228ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd228ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd228ff9640 10 monclient(hunting): my global_id is 4528 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.278+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd218004eb0 con 0x7fd22c0a9d70 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 10 client.?.objecter ms_handle_connect 0x7fd22c1457f0 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093897821 0 0) 0x7fd22c0ac400 con 0x7fd22c1457f0 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 10 monclient(hunting): my global_id is 4631 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd22c13c000 con 0x7fd22c1457f0 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1485588065 0 0) 0x7fd218004eb0 con 0x7fd22c0a9d70 2024-03-29T10:43:58.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.279+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd22c0ac400 con 0x7fd22c0a9d70 2024-03-29T10:43:58.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd21c0017d0 con 0x7fd22c0a9d70 2024-03-29T10:43:58.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:58.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:58.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 monclient(hunting): dump: 2024-03-29T10:43:58.279 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:58.279 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:58.279 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.279 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.279 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:58.279 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4187066884 0 0) 0x7fd22c13c000 con 0x7fd22c1457f0 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd218004eb0 con 0x7fd22c1457f0 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd224001c30 con 0x7fd22c1457f0 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 monclient(hunting): dump: 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:58.280 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 client.?.objecter ms_handle_connect 0x7fd22c13f650 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3541617877 0 0) 0x7fd218004eb0 con 0x7fd22c1457f0 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 >> v1:172.21.15.136:6789/0 conn(0x7fd22c0a9d70 legacy=0x7fd22c070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 >> v1:172.21.15.117:6789/0 conn(0x7fd22c13f650 legacy=0x7fd22c071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:58.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.280+0000 7fd228ff9640 1 monclient: found mon.c 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0ac6e0 con 0x7fd22c1457f0 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 10 monclient: _finish_auth 0 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd23297c640 5 monclient: authenticate success, global_id 4631 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd23297c640 10 monclient: _renew_subs 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd23297c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c0b1600 con 0x7fd22c1457f0 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd23297c640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd23297c640 10 monclient: _renew_subs 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd23297c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c06f520 con 0x7fd22c1457f0 2024-03-29T10:43:58.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd224002080 con 0x7fd22c1457f0 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd23297c640 1 librados: init done 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd224003b30 con 0x7fd22c1457f0 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd228ff9640 10 monclient: dump: 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:43:58.283 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.281+0000 7fd22a7fc640 4 set_mon_vals no callback set 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.282+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fd224019b00 con 0x7fd22c1457f0 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.282+0000 7fd23297c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.283+0000 7fd23297c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.283+0000 7fd23297c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.283+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c06f870 con 0x7fd22c1457f0 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.284+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fd224083a20 con 0x7fd22c1457f0 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.284+0000 7fd228ff9640 10 client.4631.objecter ms_dispatch 0x7fd22c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.284+0000 7fd228ff9640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.284+0000 7fd228ff9640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:58.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.284+0000 7fd228ff9640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T10:43:58.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.284+0000 7fd22bfff640 10 monclient: get_auth_request con 0x7fd21806dd90 auth_method 0 2024-03-29T10:43:58.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.284+0000 7fd228ff9640 10 client.4631.objecter ms_handle_connect 0x7fd21806dd90 2024-03-29T10:43:58.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.290+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd22404f360 con 0x7fd22c1457f0 2024-03-29T10:43:58.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.290+0000 7fd228ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:58.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.290+0000 7fd228ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:58.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.590+0000 7fd23297c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:58.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.590+0000 7fd23297c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:58.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.590+0000 7fd23297c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:43:58.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.590+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd22c13bd60 con 0x7fd22c1457f0 2024-03-29T10:43:58.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.591+0000 7fd228ff9640 1 -- 172.21.15.117:0/1725155785 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fd224051290 con 0x7fd22c1457f0 2024-03-29T10:43:58.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.591+0000 7fd228ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:58.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.591+0000 7fd228ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:58.575 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:43:58.575 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:58.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.593+0000 7fd23297c640 10 librados: watch_flush enter 2024-03-29T10:43:58.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.593+0000 7fd23297c640 10 librados: watch_flush exit 2024-03-29T10:43:58.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.593+0000 7fd23297c640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T10:43:58.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.593+0000 7fd23297c640 10 client.4631.objecter successfully canceled tick 2024-03-29T10:43:58.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.594+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 >> v1:172.21.15.117:6816/376768171 conn(0x7fd21806dd90 legacy=0x7fd218070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:58.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.594+0000 7fd23297c640 10 monclient: shutdown 2024-03-29T10:43:58.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.594+0000 7fd23297c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:58.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.594+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 >> v1:172.21.15.117:6790/0 conn(0x7fd22c1457f0 legacy=0x7fd22c1e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:58.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.594+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 shutdown_connections 2024-03-29T10:43:58.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.594+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 >> 172.21.15.117:0/1725155785 conn(0x7fd22c066a00 msgr2=0x7fd22c0b6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:58.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.596+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 shutdown_connections 2024-03-29T10:43:58.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.596+0000 7fd23297c640 1 -- 172.21.15.117:0/1725155785 wait complete. 2024-03-29T10:43:58.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:43:58.598+0000 7fd23297c640 1 librados: shutdown 2024-03-29T10:43:58.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:05.403 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:05.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.579+0000 7fd89009d640 10 monclient: get_monmap_and_config 2024-03-29T10:44:05.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.579+0000 7fd89009d640 10 monclient: build_initial_monmap 2024-03-29T10:44:05.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.579+0000 7fd89009d640 10 monclient: monmap: 2024-03-29T10:44:05.562 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:05.563 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:05.563 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:44:05.580263+0000 2024-03-29T10:44:05.563 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:44:05.580263+0000 2024-03-29T10:44:05.563 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:05.563 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:05.563 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:05.563 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:44:05.563 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:44:05.563 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:05.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.580+0000 7fd89009d640 1 Processor -- start 2024-03-29T10:44:05.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.580+0000 7fd89009d640 1 -- start start 2024-03-29T10:44:05.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.580+0000 7fd89009d640 10 monclient: init 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 10 monclient(hunting): picked mon.noname-c con 0x7fd8881422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 10 monclient(hunting): picked mon.noname-a con 0x7fd888142690 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 10 monclient(hunting): picked mon.noname-b con 0x7fd8880b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd888142c80 con 0x7fd888142690 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8880aae40 con 0x7fd8881422b0 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88813fbd0 con 0x7fd8880b17d0 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd89009d640 10 monclient(hunting): authenticate will time out at 1366.843393s 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd88d638640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd888142690 0x7fd8880b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:42690/0 (socket says 172.21.15.117:42690) 2024-03-29T10:44:05.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.581+0000 7fd88d638640 1 -- 172.21.15.117:0/2171851161 learned_addr learned my addr 172.21.15.117:0/2171851161 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:05.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517212096 0 0) 0x7fd888142c80 con 0x7fd888142690 2024-03-29T10:44:05.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): my global_id is 4563 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd874003520 con 0x7fd888142690 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192729069 0 0) 0x7fd88813fbd0 con 0x7fd8880b17d0 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd888142c80 con 0x7fd8880b17d0 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032690392 0 0) 0x7fd8880aae40 con 0x7fd8881422b0 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 10 monclient(hunting): my global_id is 4531 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.582+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd88813fbd0 con 0x7fd8881422b0 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4220446754 0 0) 0x7fd888142c80 con 0x7fd8880b17d0 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 >> v1:172.21.15.136:6789/0 conn(0x7fd8881422b0 legacy=0x7fd88813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 >> v1:172.21.15.117:6789/0 conn(0x7fd888142690 legacy=0x7fd8880b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 10 monclient: _finish_hunting 0 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 1 monclient: found mon.noname-b 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd888151940 con 0x7fd8880b17d0 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 10 monclient: _finish_auth 0 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 10 monclient: _check_auth_tickets 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:05.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd89009d640 5 monclient: authenticate success, global_id 4634 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd89009d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd884001670 con 0x7fd8880b17d0 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 10 monclient: dump: 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:05.572 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:05.573 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:05.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd89009d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:05.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd884001890 con 0x7fd8880b17d0 2024-03-29T10:44:05.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd88ce37640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:05.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd89009d640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:05.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.583+0000 7fd89009d640 4 set_mon_vals no callback set 2024-03-29T10:44:05.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.584+0000 7fd88ce37640 1 -- 172.21.15.117:0/2171851161 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd884001ba0 con 0x7fd8880b17d0 2024-03-29T10:44:05.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.584+0000 7fd88ce37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.584+0000 7fd88ce37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:05.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.584+0000 7fd88ce37640 10 monclient: dump: 2024-03-29T10:44:05.574 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:05.574 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:05.574 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.574 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.574 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.574 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.584+0000 7fd89009d640 10 monclient: shutdown 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.584+0000 7fd89009d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.584+0000 7fd89009d640 1 -- 172.21.15.117:0/2171851161 >> v1:172.21.15.117:6790/0 conn(0x7fd8880b17d0 legacy=0x7fd8880b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.584+0000 7fd89009d640 1 -- 172.21.15.117:0/2171851161 shutdown_connections 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.584+0000 7fd89009d640 1 -- 172.21.15.117:0/2171851161 >> 172.21.15.117:0/2171851161 conn(0x7fd888066a00 msgr2=0x7fd888066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.585+0000 7fd89009d640 1 -- 172.21.15.117:0/2171851161 shutdown_connections 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.585+0000 7fd89009d640 1 -- 172.21.15.117:0/2171851161 wait complete. 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.586+0000 7fd89009d640 10 monclient: build_initial_monmap 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.586+0000 7fd89009d640 10 monclient: monmap: 2024-03-29T10:44:05.575 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:05.576 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:05.576 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:44:05.576 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:44:05.576 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:05.576 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:05.576 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:05.576 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:44:05.576 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:44:05.576 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:05.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.586+0000 7fd89009d640 1 librados: starting msgr at 2024-03-29T10:44:05.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.586+0000 7fd89009d640 1 librados: starting objecter 2024-03-29T10:44:05.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.587+0000 7fd89009d640 1 Processor -- start 2024-03-29T10:44:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.587+0000 7fd89009d640 1 -- start start 2024-03-29T10:44:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.587+0000 7fd89009d640 1 librados: setting wanted keys 2024-03-29T10:44:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.587+0000 7fd89009d640 1 librados: calling monclient init 2024-03-29T10:44:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.587+0000 7fd89009d640 10 monclient: init 2024-03-29T10:44:05.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 10 monclient(hunting): picked mon.noname-c con 0x7fd8880b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 10 monclient(hunting): picked mon.noname-b con 0x7fd8881422b0 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 10 monclient(hunting): picked mon.noname-a con 0x7fd888142690 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88806f0b0 con 0x7fd888142690 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8881322f0 con 0x7fd8881422b0 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8881316e0 con 0x7fd8880b17d0 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd89009d640 10 monclient(hunting): authenticate will time out at 1366.850683s 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd88de39640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd8880b17d0 0x7fd888130e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:46508/0 (socket says 172.21.15.117:46508) 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.588+0000 7fd88de39640 1 -- 172.21.15.117:0/3674363350 learned_addr learned my addr 172.21.15.117:0/3674363350 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:05.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.589+0000 7fd87e7fc640 10 client.?.objecter ms_handle_connect 0x7fd8880b17d0 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.589+0000 7fd87e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.589+0000 7fd87e7fc640 10 client.?.objecter ms_handle_connect 0x7fd8881422b0 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.589+0000 7fd87e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.589+0000 7fd87e7fc640 10 client.?.objecter ms_handle_connect 0x7fd888142690 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675724416 0 0) 0x7fd8881316e0 con 0x7fd8880b17d0 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): my global_id is 4637 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd86c004eb0 con 0x7fd8880b17d0 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690289271 0 0) 0x7fd88806f0b0 con 0x7fd888142690 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8881316e0 con 0x7fd888142690 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032661428 0 0) 0x7fd8881322f0 con 0x7fd8881422b0 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): my global_id is 4534 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd88806f0b0 con 0x7fd8881422b0 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1027631806 0 0) 0x7fd8881316e0 con 0x7fd888142690 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8881322f0 con 0x7fd888142690 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd884001530 con 0x7fd888142690 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:05.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 8828938 0 0) 0x7fd86c004eb0 con 0x7fd8880b17d0 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8881316e0 con 0x7fd8880b17d0 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd8780019b0 con 0x7fd8880b17d0 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.590+0000 7fd87e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:05.591 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3188470326 0 0) 0x7fd88806f0b0 con 0x7fd8881422b0 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd86c004eb0 con 0x7fd8881422b0 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd880001a30 con 0x7fd8881422b0 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.592 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604506443 0 0) 0x7fd8881316e0 con 0x7fd8880b17d0 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 >> v1:172.21.15.136:6789/0 conn(0x7fd8881422b0 legacy=0x7fd888131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 >> v1:172.21.15.117:6789/0 conn(0x7fd888142690 legacy=0x7fd88812ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 1 monclient: found mon.c 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88806de90 con 0x7fd8880b17d0 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:05.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd87e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd89009d640 5 monclient: authenticate success, global_id 4637 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd89009d640 10 monclient: _renew_subs 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd89009d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8880b6a40 con 0x7fd8880b17d0 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd89009d640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd89009d640 10 monclient: _renew_subs 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd89009d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88812f2c0 con 0x7fd8880b17d0 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.591+0000 7fd89009d640 1 librados: init done 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.592+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd878001b10 con 0x7fd8880b17d0 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.593+0000 7fd87e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.593+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fd878003a60 con 0x7fd8880b17d0 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.593+0000 7fd87e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.593+0000 7fd87e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.593+0000 7fd87e7fc640 10 monclient: dump: 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.593+0000 7fd87f7fe640 4 set_mon_vals no callback set 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.593+0000 7fd89009d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.594+0000 7fd89009d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.594+0000 7fd89009d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.594+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd870005080 con 0x7fd8880b17d0 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.594+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fd878003bc0 con 0x7fd8880b17d0 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.595+0000 7fd88d638640 10 monclient: get_auth_request con 0x7fd86c06e1f0 auth_method 0 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.595+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7fd878084b10 con 0x7fd8880b17d0 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.595+0000 7fd87e7fc640 10 client.4637.objecter ms_dispatch 0x7fd888000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.595+0000 7fd87e7fc640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.595+0000 7fd87e7fc640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.595+0000 7fd87e7fc640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.596+0000 7fd87e7fc640 10 client.4637.objecter ms_handle_connect 0x7fd86c06e1f0 2024-03-29T10:44:05.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.604+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd878050220 con 0x7fd8880b17d0 2024-03-29T10:44:05.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.605+0000 7fd87e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:05.597 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.605+0000 7fd87e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:05.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.983+0000 7fd89009d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:05.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.983+0000 7fd89009d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:05.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.983+0000 7fd89009d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:05.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.983+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd870005330 con 0x7fd8880b17d0 2024-03-29T10:44:05.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.985+0000 7fd87e7fc640 1 -- 172.21.15.117:0/3674363350 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7fd878052320 con 0x7fd8880b17d0 2024-03-29T10:44:05.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.985+0000 7fd87e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:05.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.985+0000 7fd87e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:05.968 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:44:05.969 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.988+0000 7fd89009d640 10 librados: watch_flush enter 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.988+0000 7fd89009d640 10 librados: watch_flush exit 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.988+0000 7fd89009d640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.988+0000 7fd89009d640 10 client.4637.objecter successfully canceled tick 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.989+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 >> v1:172.21.15.117:6816/376768171 conn(0x7fd86c06e1f0 legacy=0x7fd86c0706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.989+0000 7fd89009d640 10 monclient: shutdown 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.989+0000 7fd89009d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.989+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 >> v1:172.21.15.117:6790/0 conn(0x7fd8880b17d0 legacy=0x7fd888130e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.989+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 shutdown_connections 2024-03-29T10:44:05.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.989+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 >> 172.21.15.117:0/3674363350 conn(0x7fd888066a00 msgr2=0x7fd888132570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:05.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.989+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 shutdown_connections 2024-03-29T10:44:05.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.989+0000 7fd89009d640 1 -- 172.21.15.117:0/3674363350 wait complete. 2024-03-29T10:44:05.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:05.989+0000 7fd89009d640 1 librados: shutdown 2024-03-29T10:44:05.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:12.793 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.961+0000 7f7e1cd4d640 10 monclient: get_monmap_and_config 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.961+0000 7f7e1cd4d640 10 monclient: build_initial_monmap 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.961+0000 7f7e1cd4d640 10 monclient: monmap: 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:44:12.962801+0000 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:44:12.962801+0000 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:44:12.945 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:44:12.946 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.962+0000 7f7e1cd4d640 1 Processor -- start 2024-03-29T10:44:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.962+0000 7f7e1cd4d640 1 -- start start 2024-03-29T10:44:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.962+0000 7f7e1cd4d640 10 monclient: init 2024-03-29T10:44:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.963+0000 7f7e1cd4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.963+0000 7f7e1cd4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.963+0000 7f7e1cd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e180a9750 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.963+0000 7f7e1cd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e1813e1e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.963+0000 7f7e1cd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e18071c30 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:12.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1cd4d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e180ab300 con 0x7f7e18071c30 2024-03-29T10:44:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1cd4d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e180ac2f0 con 0x7f7e180a9750 2024-03-29T10:44:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1cd4d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e180ad8d0 con 0x7f7e1813e1e0 2024-03-29T10:44:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1cd4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1cd4d640 10 monclient(hunting): authenticate will time out at 1374.226249s 2024-03-29T10:44:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e15d9b640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f7e1813e1e0 0x7f7e180ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41306/0 (socket says 172.21.15.117:41306) 2024-03-29T10:44:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1659c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7e180a9750 0x7f7e1813dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:35608/0 (socket says 172.21.15.117:35608) 2024-03-29T10:44:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e15d9b640 1 -- 172.21.15.117:0/3691298740 learned_addr learned my addr 172.21.15.117:0/3691298740 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:12.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1559a640 1 -- 172.21.15.117:0/3691298740 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992621162 0 0) 0x7f7e180ab300 con 0x7f7e18071c30 2024-03-29T10:44:12.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1559a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:12.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:12.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:12.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1559a640 10 monclient(hunting): my global_id is 4569 2024-03-29T10:44:12.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.964+0000 7f7e1559a640 1 -- 172.21.15.117:0/3691298740 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df8003440 con 0x7f7e18071c30 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.967+0000 7f7e1559a640 1 -- 172.21.15.117:0/3691298740 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1555969392 0 0) 0x7f7df8003440 con 0x7f7e18071c30 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 1 -- 172.21.15.117:0/3691298740 >> v1:172.21.15.117:6790/0 conn(0x7f7e1813e1e0 legacy=0x7f7e180ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 1 -- 172.21.15.117:0/3691298740 >> v1:172.21.15.136:6789/0 conn(0x7f7e180a9750 legacy=0x7f7e1813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 1 monclient: found mon.noname-a 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 1 -- 172.21.15.117:0/3691298740 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e180ada60 con 0x7f7e18071c30 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: _finish_auth 0 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:12.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1cd4d640 5 monclient: authenticate success, global_id 4569 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1cd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 1 -- 172.21.15.117:0/3691298740 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7e10001690 con 0x7f7e18071c30 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: dump: 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:12.952 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1cd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 1 -- 172.21.15.117:0/3691298740 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e10001870 con 0x7f7e18071c30 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 1 -- 172.21.15.117:0/3691298740 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7e10001c40 con 0x7f7e18071c30 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1559a640 10 monclient: dump: 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.953 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:12.954 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:12.954 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:12.954 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:12.954 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:12.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1cd4d640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:12.954 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.968+0000 7f7e1cd4d640 4 set_mon_vals no callback set 2024-03-29T10:44:12.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.969+0000 7f7e1cd4d640 10 monclient: shutdown 2024-03-29T10:44:12.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.969+0000 7f7e1cd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:12.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.969+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/3691298740 >> v1:172.21.15.117:6789/0 conn(0x7f7e18071c30 legacy=0x7f7e180ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.969+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/3691298740 shutdown_connections 2024-03-29T10:44:12.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.969+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/3691298740 >> 172.21.15.117:0/3691298740 conn(0x7f7e1806b660 msgr2=0x7f7e1806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:12.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.969+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/3691298740 shutdown_connections 2024-03-29T10:44:12.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.969+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/3691298740 wait complete. 2024-03-29T10:44:12.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.970+0000 7f7e1cd4d640 10 monclient: build_initial_monmap 2024-03-29T10:44:12.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.970+0000 7f7e1cd4d640 10 monclient: monmap: 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.971+0000 7f7e1cd4d640 1 librados: starting msgr at 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.971+0000 7f7e1cd4d640 1 librados: starting objecter 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.971+0000 7f7e1cd4d640 1 Processor -- start 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.971+0000 7f7e1cd4d640 1 -- start start 2024-03-29T10:44:12.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.971+0000 7f7e1cd4d640 1 librados: setting wanted keys 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.971+0000 7f7e1cd4d640 1 librados: calling monclient init 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.971+0000 7f7e1cd4d640 10 monclient: init 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e18071c30 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e180a9750 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1659c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f7e18071c30 0x7f7e181387b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44952/0 (socket says 172.21.15.117:44952) 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1659c640 1 -- 172.21.15.117:0/380846947 learned_addr learned my addr 172.21.15.117:0/380846947 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e1813e1e0 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e18141530 con 0x7f7e1813e1e0 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 10 client.?.objecter ms_handle_connect 0x7f7e18071c30 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1806e6c0 con 0x7f7e180a9750 2024-03-29T10:44:12.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e18141a10 con 0x7f7e18071c30 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 10 client.?.objecter ms_handle_connect 0x7f7e1813e1e0 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e1cd4d640 10 monclient(hunting): authenticate will time out at 1374.234848s 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587109535 0 0) 0x7f7e18141530 con 0x7f7e1813e1e0 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 10 monclient(hunting): my global_id is 4572 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.972+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dfc004eb0 con 0x7f7e1813e1e0 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866301939 0 0) 0x7f7e18141a10 con 0x7f7e18071c30 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:12.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e18141530 con 0x7f7e18071c30 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821838979 0 0) 0x7f7dfc004eb0 con 0x7f7e1813e1e0 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e18141a10 con 0x7f7e1813e1e0 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7e10001c40 con 0x7f7e1813e1e0 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 monclient(hunting): dump: 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:12.962 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 client.?.objecter ms_handle_connect 0x7f7e180a9750 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.973+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4041436007 0 0) 0x7f7e18141530 con 0x7f7e18071c30 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dfc004eb0 con 0x7f7e18071c30 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7e08001930 con 0x7f7e18071c30 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient(hunting): dump: 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:12.963 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065078287 0 0) 0x7f7e1806e6c0 con 0x7f7e180a9750 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient(hunting): my global_id is 4537 2024-03-29T10:44:12.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e18141530 con 0x7f7e180a9750 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 797518483 0 0) 0x7f7e18141a10 con 0x7f7e1813e1e0 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 >> v1:172.21.15.117:6790/0 conn(0x7f7e18071c30 legacy=0x7f7e181387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 >> v1:172.21.15.136:6789/0 conn(0x7f7e180a9750 legacy=0x7f7e18141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 1 monclient: found mon.a 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e18140a10 con 0x7f7e1813e1e0 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient: _finish_auth 0 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e1cd4d640 5 monclient: authenticate success, global_id 4572 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e1cd4d640 10 monclient: _renew_subs 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e1cd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:12.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e18140c60 con 0x7f7e1813e1e0 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e1cd4d640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e1cd4d640 10 monclient: _renew_subs 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e1cd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e18138010 con 0x7f7e1813e1e0 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.974+0000 7f7e1cd4d640 1 librados: init done 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.975+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e10003960 con 0x7f7e1813e1e0 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.975+0000 7f7e0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.975+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f7e10001250 con 0x7f7e1813e1e0 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.975+0000 7f7e0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.975+0000 7f7e0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.975+0000 7f7e0effd640 10 monclient: dump: 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:12.966 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:12.967 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.967 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:12.967 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.967 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:12.967 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:12.967 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:12.967 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:12.967 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:12.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.975+0000 7f7e14d99640 4 set_mon_vals no callback set 2024-03-29T10:44:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.976+0000 7f7e1cd4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.976+0000 7f7e1cd4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.976+0000 7f7e1cd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.976+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df0005080 con 0x7f7e1813e1e0 2024-03-29T10:44:12.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.977+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f7e10019ae0 con 0x7f7e1813e1e0 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.978+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4118349536 0 0) 0x7f7e10083990 con 0x7f7e1813e1e0 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.978+0000 7f7e0effd640 10 client.4572.objecter ms_dispatch 0x7f7e18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.978+0000 7f7e0effd640 3 client.4572.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.978+0000 7f7e0effd640 3 client.4572.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.978+0000 7f7e0effd640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.978+0000 7f7e1659c640 10 monclient: get_auth_request con 0x7f7dfc0767a0 auth_method 0 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.978+0000 7f7e0effd640 10 client.4572.objecter ms_handle_connect 0x7f7dfc0767a0 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.984+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e1004f0a0 con 0x7f7e1813e1e0 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.984+0000 7f7e0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:12.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:12.984+0000 7f7e0effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:13.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.313+0000 7f7e1cd4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:13.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.313+0000 7f7e1cd4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:13.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.313+0000 7f7e1cd4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:13.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.313+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7df0005330 con 0x7f7e1813e1e0 2024-03-29T10:44:13.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.314+0000 7f7e0effd640 1 -- 172.21.15.117:0/380846947 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3745307628) 0x7f7e100511a0 con 0x7f7e1813e1e0 2024-03-29T10:44:13.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.314+0000 7f7e0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:13.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.314+0000 7f7e0effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:13.298 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:44:13.298 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:34:17.050651+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:17.050651+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 10 librados: watch_flush enter 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 10 librados: watch_flush exit 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 10 client.4572.objecter successfully canceled tick 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 >> v1:172.21.15.117:6816/376768171 conn(0x7f7dfc0767a0 legacy=0x7f7dfc078c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 10 monclient: shutdown 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 >> v1:172.21.15.117:6789/0 conn(0x7f7e1813e1e0 legacy=0x7f7e181e2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 shutdown_connections 2024-03-29T10:44:13.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 >> 172.21.15.117:0/380846947 conn(0x7f7e1806b660 msgr2=0x7f7e1813ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:13.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 shutdown_connections 2024-03-29T10:44:13.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 1 -- 172.21.15.117:0/380846947 wait complete. 2024-03-29T10:44:13.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:13.318+0000 7f7e1cd4d640 1 librados: shutdown 2024-03-29T10:44:13.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:20.122 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:20.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.290+0000 7ffb2033d640 10 monclient: get_monmap_and_config 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.290+0000 7ffb2033d640 10 monclient: build_initial_monmap 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.290+0000 7ffb2033d640 10 monclient: monmap: 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:44:20.291753+0000 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:44:20.291753+0000 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:44:20.274 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:20.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.291+0000 7ffb2033d640 1 Processor -- start 2024-03-29T10:44:20.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.293+0000 7ffb2033d640 1 -- start start 2024-03-29T10:44:20.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.293+0000 7ffb2033d640 10 monclient: init 2024-03-29T10:44:20.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.294+0000 7ffb2033d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:20.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.294+0000 7ffb2033d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:20.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.294+0000 7ffb2033d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb10006a90 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:20.277 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.294+0000 7ffb2033d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb1002b120 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.294+0000 7ffb2033d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb10029c50 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb2033d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1002a290 con 0x7ffb10029c50 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb1e0d9640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ffb10006a90 0x7ffb100c8d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:45795/0 (socket says 172.21.15.117:45795) 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb1e0d9640 1 -- 172.21.15.117:0/582192973 learned_addr learned my addr 172.21.15.117:0/582192973 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb2033d640 1 -- 172.21.15.117:0/582192973 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1002c460 con 0x7ffb1002b120 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb2033d640 1 -- 172.21.15.117:0/582192973 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb100c9460 con 0x7ffb10006a90 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb2033d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb2033d640 10 monclient(hunting): authenticate will time out at 1381.559906s 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153096287 0 0) 0x7ffb1002c460 con 0x7ffb1002b120 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb1d0d7640 10 monclient(hunting): _negotiate 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb1d0d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb1d0d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:20.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.297+0000 7ffb1d0d7640 10 monclient(hunting): my global_id is 4540 2024-03-29T10:44:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04003520 con 0x7ffb1002b120 2024-03-29T10:44:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682357939 0 0) 0x7ffb100c9460 con 0x7ffb10006a90 2024-03-29T10:44:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 10 monclient(hunting): _negotiate 2024-03-29T10:44:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:44:20.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1002c460 con 0x7ffb10006a90 2024-03-29T10:44:20.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837532791 0 0) 0x7ffb1002a290 con 0x7ffb10029c50 2024-03-29T10:44:20.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 10 monclient(hunting): _negotiate 2024-03-29T10:44:20.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 10 monclient(hunting): my global_id is 4575 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.298+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb100c9460 con 0x7ffb10029c50 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 649567961 0 0) 0x7ffb1002c460 con 0x7ffb10006a90 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 >> v1:172.21.15.136:6789/0 conn(0x7ffb1002b120 legacy=0x7ffb100cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 >> v1:172.21.15.117:6789/0 conn(0x7ffb10029c50 legacy=0x7ffb100d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: _finish_hunting 0 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 1 monclient: found mon.noname-b 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb100c95f0 con 0x7ffb10006a90 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: _finish_auth 0 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: _check_auth_tickets 2024-03-29T10:44:20.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ffb14001670 con 0x7ffb10006a90 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: dump: 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:20.286 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb140017d0 con 0x7ffb10006a90 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 1 -- 172.21.15.117:0/582192973 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ffb14001c20 con 0x7ffb10006a90 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.299+0000 7ffb1d0d7640 10 monclient: dump: 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:20.287 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:20.288 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:20.288 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.301+0000 7ffb2033d640 5 monclient: authenticate success, global_id 4643 2024-03-29T10:44:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.301+0000 7ffb2033d640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.301+0000 7ffb2033d640 4 set_mon_vals no callback set 2024-03-29T10:44:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.302+0000 7ffb2033d640 10 monclient: shutdown 2024-03-29T10:44:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.302+0000 7ffb2033d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:20.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.302+0000 7ffb2033d640 1 -- 172.21.15.117:0/582192973 >> v1:172.21.15.117:6790/0 conn(0x7ffb10006a90 legacy=0x7ffb100c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:20.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.303+0000 7ffb2033d640 1 -- 172.21.15.117:0/582192973 shutdown_connections 2024-03-29T10:44:20.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.303+0000 7ffb2033d640 1 -- 172.21.15.117:0/582192973 >> 172.21.15.117:0/582192973 conn(0x7ffb10005910 msgr2=0x7ffb10005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:20.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.306+0000 7ffb2033d640 1 -- 172.21.15.117:0/582192973 shutdown_connections 2024-03-29T10:44:20.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.306+0000 7ffb2033d640 1 -- 172.21.15.117:0/582192973 wait complete. 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.307+0000 7ffb2033d640 10 monclient: build_initial_monmap 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.307+0000 7ffb2033d640 10 monclient: monmap: 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:44:20.290 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:44:20.291 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:20.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.308+0000 7ffb2033d640 1 librados: starting msgr at 2024-03-29T10:44:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.308+0000 7ffb2033d640 1 librados: starting objecter 2024-03-29T10:44:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.308+0000 7ffb2033d640 1 Processor -- start 2024-03-29T10:44:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.308+0000 7ffb2033d640 1 -- start start 2024-03-29T10:44:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.308+0000 7ffb2033d640 1 librados: setting wanted keys 2024-03-29T10:44:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.308+0000 7ffb2033d640 1 librados: calling monclient init 2024-03-29T10:44:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.308+0000 7ffb2033d640 10 monclient: init 2024-03-29T10:44:20.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.309+0000 7ffb2033d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.309+0000 7ffb2033d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.309+0000 7ffb2033d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb10006a90 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:20.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.310+0000 7ffb2033d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb10029c50 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:20.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.310+0000 7ffb2033d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb100c2d20 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:20.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.311+0000 7ffb2033d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1002eff0 con 0x7ffb100c2d20 2024-03-29T10:44:20.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.311+0000 7ffb2033d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1002f2d0 con 0x7ffb10006a90 2024-03-29T10:44:20.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.311+0000 7ffb2033d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1002e140 con 0x7ffb10029c50 2024-03-29T10:44:20.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.311+0000 7ffb2033d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:20.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.311+0000 7ffb2033d640 10 monclient(hunting): authenticate will time out at 1381.573171s 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.312+0000 7ffb1e0d9640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ffb10006a90 0x7ffb100c2610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:56924/0 (socket says 172.21.15.117:56924) 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.312+0000 7ffb1e0d9640 1 -- 172.21.15.117:0/2454362039 learned_addr learned my addr 172.21.15.117:0/2454362039 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 10 client.?.objecter ms_handle_connect 0x7ffb10029c50 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837731071 0 0) 0x7ffb1002e140 con 0x7ffb10029c50 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffafc004eb0 con 0x7ffb10029c50 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 10 client.?.objecter ms_handle_connect 0x7ffb10006a90 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3819492734 0 0) 0x7ffafc004eb0 con 0x7ffb10029c50 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb1002e140 con 0x7ffb10029c50 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.313+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ffb08001780 con 0x7ffb10029c50 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient(hunting): dump: 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:20.298 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209720807 0 0) 0x7ffb1002f2d0 con 0x7ffb10006a90 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient(hunting): my global_id is 4543 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffafc004eb0 con 0x7ffb10006a90 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2390193700 0 0) 0x7ffb1002e140 con 0x7ffb10029c50 2024-03-29T10:44:20.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 >> v1:172.21.15.136:6789/0 conn(0x7ffb10006a90 legacy=0x7ffb100c2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 >> v1:172.21.15.117:6789/0 conn(0x7ffb100c2d20 legacy=0x7ffb1016c1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 1 monclient: found mon.c 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1002ce70 con 0x7ffb10029c50 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient: _finish_auth 0 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb2033d640 5 monclient: authenticate success, global_id 4646 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb2033d640 10 monclient: _renew_subs 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb08001d30 con 0x7ffb10029c50 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb2033d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1002d120 con 0x7ffb10029c50 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb2033d640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb2033d640 10 monclient: _renew_subs 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb2033d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1002dc60 con 0x7ffb10029c50 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.314+0000 7ffb2033d640 1 librados: init done 2024-03-29T10:44:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.315+0000 7ffb0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:20.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.315+0000 7ffb0ffff640 4 set_mon_vals no callback set 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.316+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7ffb08003790 con 0x7ffb10029c50 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.316+0000 7ffb0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.316+0000 7ffb0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.316+0000 7ffb0effd640 10 monclient: dump: 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:20.303 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.317+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7ffb08019760 con 0x7ffb10029c50 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.318+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1532302323 0 0) 0x7ffb08083760 con 0x7ffb10029c50 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.318+0000 7ffb0effd640 10 client.4646.objecter ms_dispatch 0x7ffb18000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.318+0000 7ffb0effd640 3 client.4646.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.318+0000 7ffb0effd640 3 client.4646.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.318+0000 7ffb0effd640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.319+0000 7ffb1e0d9640 10 monclient: get_auth_request con 0x7ffafc06dc90 auth_method 0 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.319+0000 7ffb2033d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.319+0000 7ffb2033d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.319+0000 7ffb2033d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.319+0000 7ffb0effd640 10 client.4646.objecter ms_handle_connect 0x7ffafc06dc90 2024-03-29T10:44:20.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.319+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb00005080 con 0x7ffb10029c50 2024-03-29T10:44:20.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.327+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb0804f040 con 0x7ffb10029c50 2024-03-29T10:44:20.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.327+0000 7ffb0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:20.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.327+0000 7ffb0effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:20.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.651+0000 7ffb2033d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:20.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.651+0000 7ffb2033d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:20.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.651+0000 7ffb2033d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:20.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.651+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb00005330 con 0x7ffb10029c50 2024-03-29T10:44:20.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.653+0000 7ffb0effd640 1 -- 172.21.15.117:0/2454362039 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2172377973) 0x7ffb08050f70 con 0x7ffb10029c50 2024-03-29T10:44:20.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.653+0000 7ffb0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:20.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.653+0000 7ffb0effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:20.636 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:44:20.636 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:44:19.785478+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:19.785478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:20.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.656+0000 7ffb2033d640 10 librados: watch_flush enter 2024-03-29T10:44:20.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.657+0000 7ffb2033d640 10 librados: watch_flush exit 2024-03-29T10:44:20.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.657+0000 7ffb2033d640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T10:44:20.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.657+0000 7ffb2033d640 10 client.4646.objecter successfully canceled tick 2024-03-29T10:44:20.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.657+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 >> v1:172.21.15.117:6816/376768171 conn(0x7ffafc06dc90 legacy=0x7ffafc070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:20.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.657+0000 7ffb2033d640 10 monclient: shutdown 2024-03-29T10:44:20.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.657+0000 7ffb2033d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:20.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.657+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 >> v1:172.21.15.117:6790/0 conn(0x7ffb10029c50 legacy=0x7ffb10160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:20.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.657+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 shutdown_connections 2024-03-29T10:44:20.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.657+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 >> 172.21.15.117:0/2454362039 conn(0x7ffb10005910 msgr2=0x7ffb100c7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:20.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.658+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 shutdown_connections 2024-03-29T10:44:20.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.658+0000 7ffb2033d640 1 -- 172.21.15.117:0/2454362039 wait complete. 2024-03-29T10:44:20.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:20.658+0000 7ffb2033d640 1 librados: shutdown 2024-03-29T10:44:20.658 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:27.465 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:27.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.636+0000 7f2401948640 10 monclient: get_monmap_and_config 2024-03-29T10:44:27.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.636+0000 7f2401948640 10 monclient: build_initial_monmap 2024-03-29T10:44:27.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.636+0000 7f2401948640 10 monclient: monmap: 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:44:27.637951+0000 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:44:27.637951+0000 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:44:27.623 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:27.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.640+0000 7f2401948640 1 Processor -- start 2024-03-29T10:44:27.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.643+0000 7f2401948640 1 -- start start 2024-03-29T10:44:27.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.643+0000 7f2401948640 10 monclient: init 2024-03-29T10:44:27.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:27.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:27.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 10 monclient(hunting): picked mon.noname-a con 0x7f23fc0b4100 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:27.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 10 monclient(hunting): picked mon.noname-b con 0x7f23fc13ee90 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:27.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 10 monclient(hunting): picked mon.noname-c con 0x7f23fc0af900 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:27.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc0afd10 con 0x7f23fc0b4100 2024-03-29T10:44:27.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc0af6f0 con 0x7f23fc0af900 2024-03-29T10:44:27.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc0aaf60 con 0x7f23fc13ee90 2024-03-29T10:44:27.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:27.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.644+0000 7f2401948640 10 monclient(hunting): authenticate will time out at 1388.907534s 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.648+0000 7f23fa7fc640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f23fc13ee90 0x7f23fc0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44851/0 (socket says 172.21.15.117:44851) 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.648+0000 7f23fa7fc640 1 -- 172.21.15.117:0/1243175400 learned_addr learned my addr 172.21.15.117:0/1243175400 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.649+0000 7f23f9ffb640 1 -- 172.21.15.117:0/1243175400 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223786886 0 0) 0x7f23fc0aaf60 con 0x7f23fc13ee90 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.649+0000 7f23f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.649+0000 7f23f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.649+0000 7f23f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.649+0000 7f23f9ffb640 10 monclient(hunting): my global_id is 4649 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.649+0000 7f23f9ffb640 1 -- 172.21.15.117:0/1243175400 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e4003520 con 0x7f23fc13ee90 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 1 -- 172.21.15.117:0/1243175400 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3660390119 0 0) 0x7f23e4003520 con 0x7f23fc13ee90 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 1 -- 172.21.15.117:0/1243175400 >> v1:172.21.15.136:6789/0 conn(0x7f23fc0af900 legacy=0x7f23fc13b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:44:27.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 1 -- 172.21.15.117:0/1243175400 >> v1:172.21.15.117:6789/0 conn(0x7f23fc0b4100 legacy=0x7f23fc0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 1 monclient: found mon.noname-b 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 1 -- 172.21.15.117:0/1243175400 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc13e970 con 0x7f23fc13ee90 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 1 -- 172.21.15.117:0/1243175400 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f23f0001690 con 0x7f23fc13ee90 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 10 monclient: dump: 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:27.635 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.636 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.636 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.636 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:27.636 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:27.636 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:27.636 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:27.636 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:27.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f2401948640 5 monclient: authenticate success, global_id 4649 2024-03-29T10:44:27.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f2401948640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:27.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 1 -- 172.21.15.117:0/1243175400 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23f00018f0 con 0x7f23fc13ee90 2024-03-29T10:44:27.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f23f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:27.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f2401948640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:27.637 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.650+0000 7f2401948640 4 set_mon_vals no callback set 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.651+0000 7f2401948640 10 monclient: shutdown 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.651+0000 7f2401948640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.651+0000 7f2401948640 1 -- 172.21.15.117:0/1243175400 >> v1:172.21.15.117:6790/0 conn(0x7f23fc13ee90 legacy=0x7f23fc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.652+0000 7f2401948640 1 -- 172.21.15.117:0/1243175400 shutdown_connections 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.652+0000 7f2401948640 1 -- 172.21.15.117:0/1243175400 >> 172.21.15.117:0/1243175400 conn(0x7f23fc0b35e0 msgr2=0x7f23fc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.652+0000 7f2401948640 1 -- 172.21.15.117:0/1243175400 shutdown_connections 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.652+0000 7f2401948640 1 -- 172.21.15.117:0/1243175400 wait complete. 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.653+0000 7f2401948640 10 monclient: build_initial_monmap 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.653+0000 7f2401948640 10 monclient: monmap: 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:44:27.638 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:27.639 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:27.639 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:27.639 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:44:27.639 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:44:27.639 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:27.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.653+0000 7f2401948640 1 librados: starting msgr at 2024-03-29T10:44:27.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.653+0000 7f2401948640 1 librados: starting objecter 2024-03-29T10:44:27.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.653+0000 7f2401948640 1 Processor -- start 2024-03-29T10:44:27.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.653+0000 7f2401948640 1 -- start start 2024-03-29T10:44:27.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.653+0000 7f2401948640 1 librados: setting wanted keys 2024-03-29T10:44:27.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.653+0000 7f2401948640 1 librados: calling monclient init 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.653+0000 7f2401948640 10 monclient: init 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 10 monclient(hunting): picked mon.noname-b con 0x7f23fc0af900 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 10 monclient(hunting): picked mon.noname-a con 0x7f23fc0b4100 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 10 monclient(hunting): picked mon.noname-c con 0x7f23fc12e830 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc141f40 con 0x7f23fc0b4100 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc130fe0 con 0x7f23fc0af900 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc142190 con 0x7f23fc12e830 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.654+0000 7f2401948640 10 monclient(hunting): authenticate will time out at 1388.917861s 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.658+0000 7f23fa7fc640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f23fc0b4100 0x7f23fc131e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:45509/0 (socket says 172.21.15.117:45509) 2024-03-29T10:44:27.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.658+0000 7f23fa7fc640 1 -- 172.21.15.117:0/1145136339 learned_addr learned my addr 172.21.15.117:0/1145136339 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:27.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.661+0000 7f23e3fff640 10 client.?.objecter ms_handle_connect 0x7f23fc0b4100 2024-03-29T10:44:27.645 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.661+0000 7f23e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:27.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.661+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743944849 0 0) 0x7f23fc141f40 con 0x7f23fc0b4100 2024-03-29T10:44:27.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.661+0000 7f23e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.661+0000 7f23e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.661+0000 7f23e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.661+0000 7f23e3fff640 10 monclient(hunting): my global_id is 4578 2024-03-29T10:44:27.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.661+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e8004eb0 con 0x7f23fc0b4100 2024-03-29T10:44:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 client.?.objecter ms_handle_connect 0x7f23fc12e830 2024-03-29T10:44:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214125941 0 0) 0x7f23e8004eb0 con 0x7f23fc0b4100 2024-03-29T10:44:27.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23fc141f40 con 0x7f23fc0b4100 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 client.?.objecter ms_handle_connect 0x7f23fc0af900 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f23f0001650 con 0x7f23fc0b4100 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): dump: 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301223313 0 0) 0x7f23fc142190 con 0x7f23fc12e830 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e8004eb0 con 0x7f23fc12e830 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639115506 0 0) 0x7f23fc130fe0 con 0x7f23fc0af900 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 10 monclient(hunting): my global_id is 4546 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.662+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23fc142190 con 0x7f23fc0af900 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3344668095 0 0) 0x7f23fc142190 con 0x7f23fc0af900 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23fc130fe0 con 0x7f23fc0af900 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3241495016 0 0) 0x7f23e8004eb0 con 0x7f23fc12e830 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23fc142190 con 0x7f23fc12e830 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f23f40019d0 con 0x7f23fc0af900 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 10 monclient(hunting): dump: 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f23ec0016b0 con 0x7f23fc12e830 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.663+0000 7f23e3fff640 10 monclient(hunting): dump: 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2296233631 0 0) 0x7f23fc130fe0 con 0x7f23fc0af900 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 >> v1:172.21.15.117:6790/0 conn(0x7f23fc12e830 legacy=0x7f23fc1ee820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 >> v1:172.21.15.117:6789/0 conn(0x7f23fc0b4100 legacy=0x7f23fc131e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 1 monclient: found mon.b 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc12f400 con 0x7f23fc0af900 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f23e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f2401948640 5 monclient: authenticate success, global_id 4546 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f2401948640 10 monclient: _renew_subs 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f2401948640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc12f850 con 0x7f23fc0af900 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f2401948640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f2401948640 10 monclient: _renew_subs 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f2401948640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc1f0120 con 0x7f23fc0af900 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.664+0000 7f2401948640 1 librados: init done 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.665+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23f4001ee0 con 0x7f23fc0af900 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.665+0000 7f23e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.665+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f23f4003b10 con 0x7f23fc0af900 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.665+0000 7f23e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.665+0000 7f23e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.665+0000 7f23e3fff640 10 monclient: dump: 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.665+0000 7f23f97fa640 4 set_mon_vals no callback set 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.666+0000 7f2401948640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.666+0000 7f2401948640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.666+0000 7f2401948640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:44:27.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.666+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23dc005080 con 0x7f23fc0af900 2024-03-29T10:44:27.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.667+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f23f4003c70 con 0x7f23fc0af900 2024-03-29T10:44:27.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.670+0000 7f23fa7fc640 10 monclient: get_auth_request con 0x7f23e806e210 auth_method 0 2024-03-29T10:44:27.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.671+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1532302323 0 0) 0x7f23f4083990 con 0x7f23fc0af900 2024-03-29T10:44:27.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.671+0000 7f23e3fff640 10 client.4546.objecter ms_dispatch 0x7f23fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:44:27.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.671+0000 7f23e3fff640 3 client.4546.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:44:27.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.671+0000 7f23e3fff640 3 client.4546.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:44:27.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.671+0000 7f23e3fff640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T10:44:27.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.671+0000 7f23e3fff640 10 client.4546.objecter ms_handle_connect 0x7f23e806e210 2024-03-29T10:44:27.661 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.678+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23f404f270 con 0x7f23fc0af900 2024-03-29T10:44:27.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.678+0000 7f23e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:27.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.678+0000 7f23e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:27.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.994+0000 7f2401948640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:27.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.994+0000 7f2401948640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:27.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.994+0000 7f2401948640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-29T10:44:27.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.994+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23dc005600 con 0x7f23fc0af900 2024-03-29T10:44:27.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.996+0000 7f23e3fff640 1 -- 172.21.15.117:0/1145136339 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2172377973) 0x7f23f40511a0 con 0x7f23fc0af900 2024-03-29T10:44:27.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.996+0000 7f23e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:27.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:27.996+0000 7f23e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:27.980 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:44:27.981 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:44:19.785478+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:19.785478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:27.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.000+0000 7f2401948640 10 librados: watch_flush enter 2024-03-29T10:44:27.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.000+0000 7f2401948640 10 librados: watch_flush exit 2024-03-29T10:44:27.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.000+0000 7f2401948640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T10:44:27.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.000+0000 7f2401948640 10 client.4546.objecter successfully canceled tick 2024-03-29T10:44:27.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.000+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 >> v1:172.21.15.117:6816/376768171 conn(0x7f23e806e210 legacy=0x7f23e80706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.000+0000 7f2401948640 10 monclient: shutdown 2024-03-29T10:44:27.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.000+0000 7f2401948640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:27.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.000+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 >> v1:172.21.15.136:6789/0 conn(0x7f23fc0af900 legacy=0x7f23fc131750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.002+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 shutdown_connections 2024-03-29T10:44:27.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.002+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 >> 172.21.15.117:0/1145136339 conn(0x7f23fc0b35e0 msgr2=0x7f23fc0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:27.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.003+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 shutdown_connections 2024-03-29T10:44:27.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.003+0000 7f2401948640 1 -- 172.21.15.117:0/1145136339 wait complete. 2024-03-29T10:44:27.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:28.003+0000 7f2401948640 1 librados: shutdown 2024-03-29T10:44:28.004 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:34.811 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:34.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.974+0000 7f76bbfff640 10 monclient: get_monmap_and_config 2024-03-29T10:44:34.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.974+0000 7f76bbfff640 10 monclient: build_initial_monmap 2024-03-29T10:44:34.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.974+0000 7f76bbfff640 10 monclient: monmap: 2024-03-29T10:44:34.957 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:34.957 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:34.957 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:44:34.975238+0000 2024-03-29T10:44:34.958 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:44:34.975238+0000 2024-03-29T10:44:34.958 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:34.958 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:34.958 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:34.958 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:44:34.958 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:44:34.958 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:34.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.975+0000 7f76bbfff640 1 Processor -- start 2024-03-29T10:44:34.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.975+0000 7f76bbfff640 1 -- start start 2024-03-29T10:44:34.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.975+0000 7f76bbfff640 10 monclient: init 2024-03-29T10:44:34.959 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:34.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:34.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f76b40389e0 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:34.960 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f76b4038dc0 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:34.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f76b4030660 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:34.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40c54c0 con 0x7f76b4030660 2024-03-29T10:44:34.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b4036430 con 0x7f76b40389e0 2024-03-29T10:44:34.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b4039840 con 0x7f76b4038dc0 2024-03-29T10:44:34.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.976+0000 7f76bbfff640 10 monclient(hunting): authenticate will time out at 1396.238544s 2024-03-29T10:44:34.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.977+0000 7f76ba7fc640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f76b4038dc0 0x7f76b40cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:43532/0 (socket says 172.21.15.117:43532) 2024-03-29T10:44:34.961 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.977+0000 7f76ba7fc640 1 -- 172.21.15.117:0/4089030986 learned_addr learned my addr 172.21.15.117:0/4089030986 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:34.964 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.978+0000 7f76b9ffb640 1 -- 172.21.15.117:0/4089030986 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203223145 0 0) 0x7f76b4039840 con 0x7f76b4038dc0 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.978+0000 7f76b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.978+0000 7f76b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.978+0000 7f76b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.978+0000 7f76b9ffb640 10 monclient(hunting): my global_id is 4655 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.978+0000 7f76b9ffb640 1 -- 172.21.15.117:0/4089030986 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a4003520 con 0x7f76b4038dc0 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.978+0000 7f76b9ffb640 1 -- 172.21.15.117:0/4089030986 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3621699709 0 0) 0x7f76a4003520 con 0x7f76b4038dc0 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.978+0000 7f76b9ffb640 1 -- 172.21.15.117:0/4089030986 >> v1:172.21.15.136:6789/0 conn(0x7f76b40389e0 legacy=0x7f76b4033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 1 -- 172.21.15.117:0/4089030986 >> v1:172.21.15.117:6789/0 conn(0x7f76b4030660 legacy=0x7f76b40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 1 monclient: found mon.noname-b 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:44:34.965 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 1 -- 172.21.15.117:0/4089030986 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b4035690 con 0x7f76b4038dc0 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 1 -- 172.21.15.117:0/4089030986 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f76b0001690 con 0x7f76b4038dc0 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.980+0000 7f76b9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.981+0000 7f76b9ffb640 10 monclient: dump: 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:34.966 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:34.967 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:34.967 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:34.967 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:34.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.981+0000 7f76bbfff640 5 monclient: authenticate success, global_id 4655 2024-03-29T10:44:34.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.981+0000 7f76bbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:34.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.983+0000 7f76b9ffb640 1 -- 172.21.15.117:0/4089030986 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76b0001860 con 0x7f76b4038dc0 2024-03-29T10:44:34.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.983+0000 7f76b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.983+0000 7f76b9ffb640 1 -- 172.21.15.117:0/4089030986 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f76b0001c70 con 0x7f76b4038dc0 2024-03-29T10:44:34.967 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.983+0000 7f76b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.983+0000 7f76b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.983+0000 7f76b9ffb640 10 monclient: dump: 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.984+0000 7f76bbfff640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:34.968 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.984+0000 7f76bbfff640 4 set_mon_vals no callback set 2024-03-29T10:44:34.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.985+0000 7f76bbfff640 10 monclient: shutdown 2024-03-29T10:44:34.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.985+0000 7f76bbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:34.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.985+0000 7f76bbfff640 1 -- 172.21.15.117:0/4089030986 >> v1:172.21.15.117:6790/0 conn(0x7f76b4038dc0 legacy=0x7f76b40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.986+0000 7f76bbfff640 1 -- 172.21.15.117:0/4089030986 shutdown_connections 2024-03-29T10:44:34.969 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.986+0000 7f76bbfff640 1 -- 172.21.15.117:0/4089030986 >> 172.21.15.117:0/4089030986 conn(0x7f76b4005c00 msgr2=0x7f76b4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:34.970 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.988+0000 7f76bbfff640 1 -- 172.21.15.117:0/4089030986 shutdown_connections 2024-03-29T10:44:34.971 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.988+0000 7f76bbfff640 1 -- 172.21.15.117:0/4089030986 wait complete. 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.990+0000 7f76bbfff640 10 monclient: build_initial_monmap 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.990+0000 7f76bbfff640 10 monclient: monmap: 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:44:34.973 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:44:34.974 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:34.974 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.991+0000 7f76bbfff640 1 librados: starting msgr at 2024-03-29T10:44:34.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.991+0000 7f76bbfff640 1 librados: starting objecter 2024-03-29T10:44:34.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.991+0000 7f76bbfff640 1 Processor -- start 2024-03-29T10:44:34.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.991+0000 7f76bbfff640 1 -- start start 2024-03-29T10:44:34.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.991+0000 7f76bbfff640 1 librados: setting wanted keys 2024-03-29T10:44:34.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.991+0000 7f76bbfff640 1 librados: calling monclient init 2024-03-29T10:44:34.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.991+0000 7f76bbfff640 10 monclient: init 2024-03-29T10:44:34.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:34.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:34.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f76b4030660 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:34.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f76b40389e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:34.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f76b40c26c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:34.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b4160c70 con 0x7f76b40c26c0 2024-03-29T10:44:34.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40c2c80 con 0x7f76b4030660 2024-03-29T10:44:34.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b4160900 con 0x7f76b40389e0 2024-03-29T10:44:34.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.992+0000 7f76bbfff640 10 monclient(hunting): authenticate will time out at 1396.254891s 2024-03-29T10:44:34.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.993+0000 7f76ba7fc640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f76b40389e0 0x7f76b40c1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:45722/0 (socket says 172.21.15.117:45722) 2024-03-29T10:44:34.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.993+0000 7f76ba7fc640 1 -- 172.21.15.117:0/1085932423 learned_addr learned my addr 172.21.15.117:0/1085932423 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:34.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.996+0000 7f76a37fe640 10 client.?.objecter ms_handle_connect 0x7f76b40389e0 2024-03-29T10:44:34.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.996+0000 7f76a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.996+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868711441 0 0) 0x7f76b4160900 con 0x7f76b40389e0 2024-03-29T10:44:34.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.996+0000 7f76a37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.996+0000 7f76a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.996+0000 7f76a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.996+0000 7f76a37fe640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:44:34.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.996+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a8004eb0 con 0x7f76b40389e0 2024-03-29T10:44:34.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 10 client.?.objecter ms_handle_connect 0x7f76b4030660 2024-03-29T10:44:34.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890162348 0 0) 0x7f76b40c2c80 con 0x7f76b4030660 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 10 monclient(hunting): my global_id is 4549 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b4160900 con 0x7f76b4030660 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 834200402 0 0) 0x7f76b4160900 con 0x7f76b4030660 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.997+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a8005490 con 0x7f76b4030660 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f76ac001810 con 0x7f76b4030660 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 10 monclient(hunting): dump: 2024-03-29T10:44:34.989 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3640757431 0 0) 0x7f76a8004eb0 con 0x7f76b40389e0 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a8005b40 con 0x7f76b40389e0 2024-03-29T10:44:34.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f76b0001bb0 con 0x7f76b40389e0 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.998+0000 7f76a37fe640 10 monclient(hunting): dump: 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4209787565 0 0) 0x7f76a8005b40 con 0x7f76b40389e0 2024-03-29T10:44:34.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 >> v1:172.21.15.136:6789/0 conn(0x7f76b4030660 legacy=0x7f76b41601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 >> v1:172.21.15.117:6789/0 conn(0x7f76b40c26c0 legacy=0x7f76b402d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 1 monclient: found mon.c 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b40c72a0 con 0x7f76b40389e0 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76b0004a90 con 0x7f76b40389e0 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:34.999+0000 7f76b97fa640 4 set_mon_vals no callback set 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.000+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f76b0004da0 con 0x7f76b40389e0 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.000+0000 7f76a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.000+0000 7f76a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.000+0000 7f76a37fe640 10 monclient: dump: 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.001+0000 7f76bbfff640 5 monclient: authenticate success, global_id 4658 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.001+0000 7f76bbfff640 10 monclient: _renew_subs 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.001+0000 7f76bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:34.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.001+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b40c7490 con 0x7f76b40389e0 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.001+0000 7f76bbfff640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.001+0000 7f76bbfff640 10 monclient: _renew_subs 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.001+0000 7f76bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.001+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b402dd70 con 0x7f76b40389e0 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.001+0000 7f76bbfff640 1 librados: init done 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.002+0000 7f76bbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.002+0000 7f76bbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.002+0000 7f76bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.002+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b4007b90 con 0x7f76b40389e0 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.003+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f76b001ad70 con 0x7f76b40389e0 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.004+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1532302323 0 0) 0x7f76b0084a60 con 0x7f76b40389e0 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.004+0000 7f76a37fe640 10 client.4658.objecter ms_dispatch 0x7f76bc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.004+0000 7f76a37fe640 3 client.4658.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:44:34.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.004+0000 7f76a37fe640 3 client.4658.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:44:34.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.004+0000 7f76a37fe640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-29T10:44:34.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.004+0000 7f76baffd640 10 monclient: get_auth_request con 0x7f76a8076a60 auth_method 0 2024-03-29T10:44:34.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.004+0000 7f76a37fe640 10 client.4658.objecter ms_handle_connect 0x7f76a8076a60 2024-03-29T10:44:34.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.012+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76b0086e60 con 0x7f76b40389e0 2024-03-29T10:44:34.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.012+0000 7f76a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.012+0000 7f76a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:35.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.343+0000 7f76bbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:35.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.343+0000 7f76bbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:35.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.343+0000 7f76bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:35.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.343+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76b402e070 con 0x7f76b40389e0 2024-03-29T10:44:35.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.345+0000 7f76a37fe640 1 -- 172.21.15.117:0/1085932423 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2172377973) 0x7f76b0086e60 con 0x7f76b40389e0 2024-03-29T10:44:35.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.345+0000 7f76a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:35.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.345+0000 7f76a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:35.329 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:44:35.329 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:44:19.785478+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:19.785478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:35.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.349+0000 7f76bbfff640 10 librados: watch_flush enter 2024-03-29T10:44:35.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.349+0000 7f76bbfff640 10 librados: watch_flush exit 2024-03-29T10:44:35.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.349+0000 7f76bbfff640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-29T10:44:35.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.349+0000 7f76bbfff640 10 client.4658.objecter successfully canceled tick 2024-03-29T10:44:35.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.349+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 >> v1:172.21.15.117:6816/376768171 conn(0x7f76a8076a60 legacy=0x7f76a8078f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.349+0000 7f76bbfff640 10 monclient: shutdown 2024-03-29T10:44:35.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.349+0000 7f76bbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.349+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 >> v1:172.21.15.117:6790/0 conn(0x7f76b40389e0 legacy=0x7f76b40c1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.350+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 shutdown_connections 2024-03-29T10:44:35.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.350+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 >> 172.21.15.117:0/1085932423 conn(0x7f76b4005c00 msgr2=0x7f76b40c3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.350+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 shutdown_connections 2024-03-29T10:44:35.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.350+0000 7f76bbfff640 1 -- 172.21.15.117:0/1085932423 wait complete. 2024-03-29T10:44:35.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:35.351+0000 7f76bbfff640 1 librados: shutdown 2024-03-29T10:44:35.351 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:42.158 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:42.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.342+0000 7f3b99cc1640 10 monclient: get_monmap_and_config 2024-03-29T10:44:42.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.342+0000 7f3b99cc1640 10 monclient: build_initial_monmap 2024-03-29T10:44:42.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.342+0000 7f3b99cc1640 10 monclient: monmap: 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:44:42.343679+0000 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:44:42.343679+0000 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:44:42.326 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:42.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.344+0000 7f3b99cc1640 1 Processor -- start 2024-03-29T10:44:42.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.344+0000 7f3b99cc1640 1 -- start start 2024-03-29T10:44:42.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.344+0000 7f3b99cc1640 10 monclient: init 2024-03-29T10:44:42.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:42.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:42.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 10 monclient(hunting): picked mon.noname-b con 0x7f3b94071de0 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:42.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 10 monclient(hunting): picked mon.noname-a con 0x7f3b940721c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:42.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 10 monclient(hunting): picked mon.noname-c con 0x7f3b94145ae0 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:42.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b94145650 con 0x7f3b940721c0 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b937fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f3b94071de0 0x7f3b940aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:54966/0 (socket says 172.21.15.117:54966) 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b937fe640 1 -- 172.21.15.117:0/1141409847 learned_addr learned my addr 172.21.15.117:0/1141409847 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 1 -- 172.21.15.117:0/1141409847 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b940ae260 con 0x7f3b94145ae0 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 1 -- 172.21.15.117:0/1141409847 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b941420d0 con 0x7f3b94071de0 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b99cc1640 10 monclient(hunting): authenticate will time out at 1403.607666s 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676269590 0 0) 0x7f3b94145650 con 0x7f3b940721c0 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.345+0000 7f3b92ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient(hunting): my global_id is 4584 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b80003520 con 0x7f3b940721c0 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202952632 0 0) 0x7f3b941420d0 con 0x7f3b94071de0 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient(hunting): my global_id is 4661 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b94145650 con 0x7f3b94071de0 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2439272442 0 0) 0x7f3b80003520 con 0x7f3b940721c0 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 >> v1:172.21.15.117:6790/0 conn(0x7f3b94071de0 legacy=0x7f3b940aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 >> v1:172.21.15.136:6789/0 conn(0x7f3b94145ae0 legacy=0x7f3b94145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 1 monclient: found mon.noname-a 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940b0590 con 0x7f3b940721c0 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.346+0000 7f3b92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3b7c001690 con 0x7f3b940721c0 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b99cc1640 5 monclient: authenticate success, global_id 4584 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b99cc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 10 monclient: dump: 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b99cc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b7c0018b0 con 0x7f3b940721c0 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:42.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 1 -- 172.21.15.117:0/1141409847 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3b7c001bc0 con 0x7f3b940721c0 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b99cc1640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b99cc1640 4 set_mon_vals no callback set 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.347+0000 7f3b92ffd640 10 monclient: dump: 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:42.333 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.348+0000 7f3b99cc1640 10 monclient: shutdown 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.348+0000 7f3b99cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.348+0000 7f3b99cc1640 1 -- 172.21.15.117:0/1141409847 >> v1:172.21.15.117:6789/0 conn(0x7f3b940721c0 legacy=0x7f3b9413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.348+0000 7f3b99cc1640 1 -- 172.21.15.117:0/1141409847 shutdown_connections 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.348+0000 7f3b99cc1640 1 -- 172.21.15.117:0/1141409847 >> 172.21.15.117:0/1141409847 conn(0x7f3b9406b660 msgr2=0x7f3b9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.348+0000 7f3b99cc1640 1 -- 172.21.15.117:0/1141409847 shutdown_connections 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.348+0000 7f3b99cc1640 1 -- 172.21.15.117:0/1141409847 wait complete. 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.349+0000 7f3b99cc1640 10 monclient: build_initial_monmap 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.349+0000 7f3b99cc1640 10 monclient: monmap: 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:42.342 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.350+0000 7f3b99cc1640 1 librados: starting msgr at 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.350+0000 7f3b99cc1640 1 librados: starting objecter 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.350+0000 7f3b99cc1640 1 Processor -- start 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.350+0000 7f3b99cc1640 1 -- start start 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.350+0000 7f3b99cc1640 1 librados: setting wanted keys 2024-03-29T10:44:42.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.350+0000 7f3b99cc1640 1 librados: calling monclient init 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.350+0000 7f3b99cc1640 10 monclient: init 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 10 monclient(hunting): picked mon.noname-b con 0x7f3b94071de0 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 10 monclient(hunting): picked mon.noname-a con 0x7f3b94145ae0 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b8bfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f3b94145ae0 0x7f3b941ded60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:34970/0 (socket says 172.21.15.117:34970) 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b8bfff640 1 -- 172.21.15.117:0/855924021 learned_addr learned my addr 172.21.15.117:0/855924021 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 10 monclient(hunting): picked mon.noname-c con 0x7f3b94144b70 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9412de10 con 0x7f3b94145ae0 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b941dfb80 con 0x7f3b94071de0 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b90ff9640 10 client.?.objecter ms_handle_connect 0x7f3b94144b70 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b90ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b94142020 con 0x7f3b94144b70 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:42.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b99cc1640 10 monclient(hunting): authenticate will time out at 1403.613873s 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b90ff9640 10 client.?.objecter ms_handle_connect 0x7f3b94145ae0 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b90ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302038971 0 0) 0x7f3b94142020 con 0x7f3b94144b70 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b90ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.351+0000 7f3b90ff9640 10 monclient(hunting): my global_id is 4664 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b74004eb0 con 0x7f3b94144b70 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296663821 0 0) 0x7f3b9412de10 con 0x7f3b94145ae0 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 10 monclient(hunting): my global_id is 4587 2024-03-29T10:44:42.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b94142020 con 0x7f3b94145ae0 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2914099629 0 0) 0x7f3b74004eb0 con 0x7f3b94144b70 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b9412de10 con 0x7f3b94144b70 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3b8c001b70 con 0x7f3b94144b70 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:42.346 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.352+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2518624246 0 0) 0x7f3b9412de10 con 0x7f3b94144b70 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 >> v1:172.21.15.136:6789/0 conn(0x7f3b94071de0 legacy=0x7f3b9406dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 >> v1:172.21.15.117:6789/0 conn(0x7f3b94145ae0 legacy=0x7f3b941ded60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 1 monclient: found mon.c 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940acc10 con 0x7f3b94144b70 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:42.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b99cc1640 5 monclient: authenticate success, global_id 4664 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b99cc1640 10 monclient: _renew_subs 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b99cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b940acef0 con 0x7f3b94144b70 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b99cc1640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b99cc1640 10 monclient: _renew_subs 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b99cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9412e080 con 0x7f3b94144b70 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b99cc1640 1 librados: init done 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b8c001cd0 con 0x7f3b94144b70 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.353+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3b8c003b10 con 0x7f3b94144b70 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.354+0000 7f3b90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.354+0000 7f3b90ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.354+0000 7f3b90ff9640 10 monclient: dump: 2024-03-29T10:44:42.348 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.354+0000 7f3b91ffb640 4 set_mon_vals no callback set 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.354+0000 7f3b99cc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.355+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f3b8c019ae0 con 0x7f3b94144b70 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.355+0000 7f3b99cc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:42.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.355+0000 7f3b99cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:42.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.355+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b78005080 con 0x7f3b94144b70 2024-03-29T10:44:42.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.356+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1532302323 0 0) 0x7f3b8c083ad0 con 0x7f3b94144b70 2024-03-29T10:44:42.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.356+0000 7f3b90ff9640 10 client.4664.objecter ms_dispatch 0x7f3b94000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:44:42.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.356+0000 7f3b937fe640 10 monclient: get_auth_request con 0x7f3b7406dad0 auth_method 0 2024-03-29T10:44:42.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.356+0000 7f3b90ff9640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:44:42.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.356+0000 7f3b90ff9640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:44:42.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.356+0000 7f3b90ff9640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-29T10:44:42.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.356+0000 7f3b90ff9640 10 client.4664.objecter ms_handle_connect 0x7f3b7406dad0 2024-03-29T10:44:42.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.364+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b8c04f3b0 con 0x7f3b94144b70 2024-03-29T10:44:42.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.364+0000 7f3b90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:42.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.364+0000 7f3b90ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.766+0000 7f3b99cc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.766+0000 7f3b99cc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.766+0000 7f3b99cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.766+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b78005330 con 0x7f3b94144b70 2024-03-29T10:44:42.750 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:44:42.750 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:44:19.785478+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:19.785478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.767+0000 7f3b90ff9640 1 -- 172.21.15.117:0/855924021 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2172377973) 0x7f3b8c0512e0 con 0x7f3b94144b70 2024-03-29T10:44:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.767+0000 7f3b90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.767+0000 7f3b90ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.771+0000 7f3b99cc1640 10 librados: watch_flush enter 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.771+0000 7f3b99cc1640 10 librados: watch_flush exit 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.771+0000 7f3b99cc1640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.771+0000 7f3b99cc1640 10 client.4664.objecter successfully canceled tick 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.771+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 >> v1:172.21.15.117:6816/376768171 conn(0x7f3b7406dad0 legacy=0x7f3b7406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.772+0000 7f3b99cc1640 10 monclient: shutdown 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.772+0000 7f3b99cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.772+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 >> v1:172.21.15.117:6790/0 conn(0x7f3b94144b70 legacy=0x7f3b941df470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.772+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 shutdown_connections 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.772+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 >> 172.21.15.117:0/855924021 conn(0x7f3b9406b660 msgr2=0x7f3b94069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.772+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 shutdown_connections 2024-03-29T10:44:42.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.772+0000 7f3b99cc1640 1 -- 172.21.15.117:0/855924021 wait complete. 2024-03-29T10:44:42.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:42.772+0000 7f3b99cc1640 1 librados: shutdown 2024-03-29T10:44:42.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:49.578 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:49.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.744+0000 7f3495fd7640 10 monclient: get_monmap_and_config 2024-03-29T10:44:49.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.744+0000 7f3495fd7640 10 monclient: build_initial_monmap 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.744+0000 7f3495fd7640 10 monclient: monmap: 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:44:49.745482+0000 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:44:49.745482+0000 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:44:49.728 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:49.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.748+0000 7f3495fd7640 1 Processor -- start 2024-03-29T10:44:49.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.748+0000 7f3495fd7640 1 -- start start 2024-03-29T10:44:49.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.748+0000 7f3495fd7640 10 monclient: init 2024-03-29T10:44:49.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 10 monclient(hunting): picked mon.noname-a con 0x7f34900b0770 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 10 monclient(hunting): picked mon.noname-c con 0x7f349014a700 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 10 monclient(hunting): picked mon.noname-b con 0x7f34900ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34900aba70 con 0x7f34900b0770 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34900aecb0 con 0x7f349014a700 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34900af150 con 0x7f34900ac5a0 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f3495fd7640 10 monclient(hunting): authenticate will time out at 1411.011271s 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f348ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f34900ac5a0 0x7f3490151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:51604/0 (socket says 172.21.15.117:51604) 2024-03-29T10:44:49.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.749+0000 7f348ffff640 1 -- 172.21.15.117:0/2604577015 learned_addr learned my addr 172.21.15.117:0/2604577015 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:49.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.750+0000 7f348e7fc640 1 -- 172.21.15.117:0/2604577015 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495373383 0 0) 0x7f34900af150 con 0x7f34900ac5a0 2024-03-29T10:44:49.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.750+0000 7f348e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:49.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.750+0000 7f348e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:49.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.750+0000 7f348e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:49.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.750+0000 7f348e7fc640 10 monclient(hunting): my global_id is 4667 2024-03-29T10:44:49.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.750+0000 7f348e7fc640 1 -- 172.21.15.117:0/2604577015 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3478003520 con 0x7f34900ac5a0 2024-03-29T10:44:49.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.753+0000 7f348e7fc640 1 -- 172.21.15.117:0/2604577015 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 749740343 0 0) 0x7f3478003520 con 0x7f34900ac5a0 2024-03-29T10:44:49.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.753+0000 7f348e7fc640 1 -- 172.21.15.117:0/2604577015 >> v1:172.21.15.136:6789/0 conn(0x7f349014a700 legacy=0x7f34900abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.753+0000 7f348e7fc640 1 -- 172.21.15.117:0/2604577015 >> v1:172.21.15.117:6789/0 conn(0x7f34900b0770 legacy=0x7f3490149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 1 -- 172.21.15.117:0/2604577015 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34900ac9d0 con 0x7f34900ac5a0 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 1 -- 172.21.15.117:0/2604577015 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3488001a10 con 0x7f34900ac5a0 2024-03-29T10:44:49.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f348e7fc640 10 monclient: dump: 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f3495fd7640 5 monclient: authenticate success, global_id 4667 2024-03-29T10:44:49.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.755+0000 7f3495fd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.756+0000 7f348e7fc640 1 -- 172.21.15.117:0/2604577015 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3488001ca0 con 0x7f34900ac5a0 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.756+0000 7f348e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.756+0000 7f3495fd7640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.756+0000 7f3495fd7640 4 set_mon_vals no callback set 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.756+0000 7f348e7fc640 1 -- 172.21.15.117:0/2604577015 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3488002030 con 0x7f34900ac5a0 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.756+0000 7f348e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.756+0000 7f348e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.756+0000 7f348e7fc640 10 monclient: dump: 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:49.742 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:49.743 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:49.743 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:49.743 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:49.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.757+0000 7f3495fd7640 10 monclient: shutdown 2024-03-29T10:44:49.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.757+0000 7f3495fd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:49.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.757+0000 7f3495fd7640 1 -- 172.21.15.117:0/2604577015 >> v1:172.21.15.117:6790/0 conn(0x7f34900ac5a0 legacy=0x7f3490151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.757+0000 7f3495fd7640 1 -- 172.21.15.117:0/2604577015 shutdown_connections 2024-03-29T10:44:49.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.757+0000 7f3495fd7640 1 -- 172.21.15.117:0/2604577015 >> 172.21.15.117:0/2604577015 conn(0x7f349006b660 msgr2=0x7f349006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:49.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.757+0000 7f3495fd7640 1 -- 172.21.15.117:0/2604577015 shutdown_connections 2024-03-29T10:44:49.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.757+0000 7f3495fd7640 1 -- 172.21.15.117:0/2604577015 wait complete. 2024-03-29T10:44:49.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.758+0000 7f3495fd7640 10 monclient: build_initial_monmap 2024-03-29T10:44:49.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.758+0000 7f3495fd7640 10 monclient: monmap: 2024-03-29T10:44:49.744 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:49.744 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:49.744 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:44:49.744 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:44:49.744 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:49.745 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:49.745 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:49.745 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:44:49.745 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:44:49.745 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:49.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.758+0000 7f3495fd7640 1 librados: starting msgr at 2024-03-29T10:44:49.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.758+0000 7f3495fd7640 1 librados: starting objecter 2024-03-29T10:44:49.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.759+0000 7f3495fd7640 1 Processor -- start 2024-03-29T10:44:49.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.759+0000 7f3495fd7640 1 -- start start 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.759+0000 7f3495fd7640 1 librados: setting wanted keys 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.759+0000 7f3495fd7640 1 librados: calling monclient init 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.759+0000 7f3495fd7640 10 monclient: init 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 10 monclient(hunting): picked mon.noname-c con 0x7f34900ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 10 monclient(hunting): picked mon.noname-b con 0x7f34900b0770 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 10 monclient(hunting): picked mon.noname-a con 0x7f349014a700 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34901db380 con 0x7f349014a700 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3490127d40 con 0x7f34900b0770 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3490125340 con 0x7f34900ac5a0 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.760+0000 7f3495fd7640 10 monclient(hunting): authenticate will time out at 1411.022419s 2024-03-29T10:44:49.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f348ffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f349014a700 0x7f34901db810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:42227/0 (socket says 172.21.15.117:42227) 2024-03-29T10:44:49.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f348ffff640 1 -- 172.21.15.117:0/2864822486 learned_addr learned my addr 172.21.15.117:0/2864822486 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:49.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f3477fff640 10 client.?.objecter ms_handle_connect 0x7f349014a700 2024-03-29T10:44:49.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f3477fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:49.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511001533 0 0) 0x7f34901db380 con 0x7f349014a700 2024-03-29T10:44:49.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f3477fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:49.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f3477fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:49.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f3477fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:49.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f3477fff640 10 monclient(hunting): my global_id is 4590 2024-03-29T10:44:49.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.763+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347c004eb0 con 0x7f349014a700 2024-03-29T10:44:49.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.767+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 494304528 0 0) 0x7f347c004eb0 con 0x7f349014a700 2024-03-29T10:44:49.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.767+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34901db380 con 0x7f349014a700 2024-03-29T10:44:49.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.767+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f34880018d0 con 0x7f349014a700 2024-03-29T10:44:49.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.767+0000 7f3477fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:49.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.767+0000 7f3477fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:49.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.767+0000 7f3477fff640 10 monclient(hunting): dump: 2024-03-29T10:44:49.752 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.767+0000 7f3477fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.767+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1586251151 0 0) 0x7f34901db380 con 0x7f349014a700 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.767+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 >> v1:172.21.15.117:6790/0 conn(0x7f34900ac5a0 legacy=0x7f3490128720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 >> v1:172.21.15.136:6789/0 conn(0x7f34900b0770 legacy=0x7f3490146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 1 monclient: found mon.a 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3490147ae0 con 0x7f349014a700 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 client.?.objecter ms_handle_connect 0x7f34900ac5a0 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 client.?.objecter ms_handle_connect 0x7f34900b0770 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3495fd7640 5 monclient: authenticate success, global_id 4590 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3495fd7640 10 monclient: _renew_subs 2024-03-29T10:44:49.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3495fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3490147cd0 con 0x7f349014a700 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3495fd7640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3495fd7640 10 monclient: _renew_subs 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3495fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34901290c0 con 0x7f349014a700 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3495fd7640 1 librados: init done 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3488001a30 con 0x7f349014a700 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f3488005110 con 0x7f349014a700 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f3477fff640 10 monclient: dump: 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:49.755 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:49.756 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:49.756 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:49.756 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:49.756 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:49.756 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:49.756 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:49.756 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:49.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.768+0000 7f348dffb640 4 set_mon_vals no callback set 2024-03-29T10:44:49.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.770+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f348801b0e0 con 0x7f349014a700 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.771+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1532302323 0 0) 0x7f3488085020 con 0x7f349014a700 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.771+0000 7f3477fff640 10 client.4590.objecter ms_dispatch 0x7f3490000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.771+0000 7f3477fff640 3 client.4590.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.771+0000 7f3477fff640 3 client.4590.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.771+0000 7f3495fd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.771+0000 7f3495fd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.771+0000 7f3495fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.771+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3470005080 con 0x7f349014a700 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.771+0000 7f3477fff640 20 client.4590.objecter dump_active .. 0 homeless 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.772+0000 7f348f7fe640 10 monclient: get_auth_request con 0x7f347c06dc30 auth_method 0 2024-03-29T10:44:49.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.772+0000 7f3477fff640 10 client.4590.objecter ms_handle_connect 0x7f347c06dc30 2024-03-29T10:44:49.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.782+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3488050730 con 0x7f349014a700 2024-03-29T10:44:49.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.782+0000 7f3477fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:49.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:49.782+0000 7f3477fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:50.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.142+0000 7f3495fd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:50.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.143+0000 7f3495fd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:50.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.143+0000 7f3495fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:44:50.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.143+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3470005330 con 0x7f349014a700 2024-03-29T10:44:50.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.145+0000 7f3477fff640 1 -- 172.21.15.117:0/2864822486 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2172377973) 0x7f3488052830 con 0x7f349014a700 2024-03-29T10:44:50.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.145+0000 7f3477fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:50.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.145+0000 7f3477fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:50.128 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:44:50.128 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:44:19.785478+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:19.785478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:50.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 10 librados: watch_flush enter 2024-03-29T10:44:50.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 10 librados: watch_flush exit 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 20 client.4590.objecter shutdown clearing up homeless session... 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 10 client.4590.objecter successfully canceled tick 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 >> v1:172.21.15.117:6816/376768171 conn(0x7f347c06dc30 legacy=0x7f347c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 10 monclient: shutdown 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 >> v1:172.21.15.117:6789/0 conn(0x7f349014a700 legacy=0x7f34901db810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 shutdown_connections 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 >> 172.21.15.117:0/2864822486 conn(0x7f349006b660 msgr2=0x7f34900abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.149+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 shutdown_connections 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.150+0000 7f3495fd7640 1 -- 172.21.15.117:0/2864822486 wait complete. 2024-03-29T10:44:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:50.150+0000 7f3495fd7640 1 librados: shutdown 2024-03-29T10:44:50.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:56.954 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:57.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.119+0000 7fcdc359e640 10 monclient: get_monmap_and_config 2024-03-29T10:44:57.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.119+0000 7fcdc359e640 10 monclient: build_initial_monmap 2024-03-29T10:44:57.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.119+0000 7fcdc359e640 10 monclient: monmap: 2024-03-29T10:44:57.103 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:57.103 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:57.103 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:44:57.120629+0000 2024-03-29T10:44:57.103 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:44:57.120629+0000 2024-03-29T10:44:57.103 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:57.104 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:57.104 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:57.104 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:44:57.104 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:44:57.104 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:57.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.121+0000 7fcdc359e640 1 Processor -- start 2024-03-29T10:44:57.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.121+0000 7fcdc359e640 1 -- start start 2024-03-29T10:44:57.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.121+0000 7fcdc359e640 10 monclient: init 2024-03-29T10:44:57.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:57.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 10 monclient(hunting): picked mon.noname-b con 0x7fcdbc006890 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 10 monclient(hunting): picked mon.noname-c con 0x7fcdbc006c70 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 10 monclient(hunting): picked mon.noname-a con 0x7fcdbc02b4c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc02c830 con 0x7fcdbc02b4c0 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc02afa0 con 0x7fcdbc006c70 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc02aba0 con 0x7fcdbc006890 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.122+0000 7fcdc359e640 10 monclient(hunting): authenticate will time out at 1418.384865s 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.123+0000 7fcdc259c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fcdbc006890 0x7fcdbc0c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:47190/0 (socket says 172.21.15.117:47190) 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.123+0000 7fcdc259c640 1 -- 172.21.15.117:0/1676701411 learned_addr learned my addr 172.21.15.117:0/1676701411 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.123+0000 7fcdc159a640 1 -- 172.21.15.117:0/1676701411 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987102976 0 0) 0x7fcdbc02aba0 con 0x7fcdbc006890 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.123+0000 7fcdc159a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.123+0000 7fcdc159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.123+0000 7fcdc159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:57.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.123+0000 7fcdc159a640 10 monclient(hunting): my global_id is 4670 2024-03-29T10:44:57.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.123+0000 7fcdc159a640 1 -- 172.21.15.117:0/1676701411 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdac003520 con 0x7fcdbc006890 2024-03-29T10:44:57.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.125+0000 7fcdc159a640 1 -- 172.21.15.117:0/1676701411 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 192523496 0 0) 0x7fcdac003520 con 0x7fcdbc006890 2024-03-29T10:44:57.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.125+0000 7fcdc159a640 1 -- 172.21.15.117:0/1676701411 >> v1:172.21.15.136:6789/0 conn(0x7fcdbc006c70 legacy=0x7fcdbc0c6d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:57.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 1 -- 172.21.15.117:0/1676701411 >> v1:172.21.15.117:6789/0 conn(0x7fcdbc02b4c0 legacy=0x7fcdbc0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 1 monclient: found mon.noname-b 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 1 -- 172.21.15.117:0/1676701411 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc02ad30 con 0x7fcdbc006890 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 10 monclient: _finish_auth 0 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 1 -- 172.21.15.117:0/1676701411 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fcdb8001690 con 0x7fcdbc006890 2024-03-29T10:44:57.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.126+0000 7fcdc159a640 10 monclient: dump: 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.127+0000 7fcdc359e640 5 monclient: authenticate success, global_id 4670 2024-03-29T10:44:57.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.127+0000 7fcdc359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.128+0000 7fcdc159a640 1 -- 172.21.15.117:0/1676701411 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdb80017f0 con 0x7fcdbc006890 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.128+0000 7fcdc159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.128+0000 7fcdc359e640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.128+0000 7fcdc359e640 4 set_mon_vals no callback set 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.129+0000 7fcdc159a640 1 -- 172.21.15.117:0/1676701411 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fcdb8001c40 con 0x7fcdbc006890 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.129+0000 7fcdc159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.129+0000 7fcdc159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.129+0000 7fcdc159a640 10 monclient: dump: 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:57.114 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:57.115 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:57.115 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:57.115 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:57.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.129+0000 7fcdc359e640 10 monclient: shutdown 2024-03-29T10:44:57.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.129+0000 7fcdc359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:57.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.129+0000 7fcdc359e640 1 -- 172.21.15.117:0/1676701411 >> v1:172.21.15.117:6790/0 conn(0x7fcdbc006890 legacy=0x7fcdbc0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:57.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.131+0000 7fcdc359e640 1 -- 172.21.15.117:0/1676701411 shutdown_connections 2024-03-29T10:44:57.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.131+0000 7fcdc359e640 1 -- 172.21.15.117:0/1676701411 >> 172.21.15.117:0/1676701411 conn(0x7fcdbc0058f0 msgr2=0x7fcdbc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:57.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.133+0000 7fcdc359e640 1 -- 172.21.15.117:0/1676701411 shutdown_connections 2024-03-29T10:44:57.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.133+0000 7fcdc359e640 1 -- 172.21.15.117:0/1676701411 wait complete. 2024-03-29T10:44:57.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.134+0000 7fcdc359e640 10 monclient: build_initial_monmap 2024-03-29T10:44:57.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.134+0000 7fcdc359e640 10 monclient: monmap: 2024-03-29T10:44:57.117 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:44:57.117 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:57.117 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:44:57.117 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:44:57.117 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:57.117 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:57.117 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:44:57.118 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:44:57.118 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:44:57.118 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:57.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.135+0000 7fcdc359e640 1 librados: starting msgr at 2024-03-29T10:44:57.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.135+0000 7fcdc359e640 1 librados: starting objecter 2024-03-29T10:44:57.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.135+0000 7fcdc359e640 1 Processor -- start 2024-03-29T10:44:57.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.135+0000 7fcdc359e640 1 -- start start 2024-03-29T10:44:57.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.135+0000 7fcdc359e640 1 librados: setting wanted keys 2024-03-29T10:44:57.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.135+0000 7fcdc359e640 1 librados: calling monclient init 2024-03-29T10:44:57.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.135+0000 7fcdc359e640 10 monclient: init 2024-03-29T10:44:57.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.136+0000 7fcdc359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:57.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.136+0000 7fcdc359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:57.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.136+0000 7fcdc359e640 10 monclient(hunting): picked mon.noname-c con 0x7fcdbc006890 addr v1:172.21.15.117:6790/0 2024-03-29T10:44:57.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdc259c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fcdbc006890 0x7fcdbc0c1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:55294/0 (socket says 172.21.15.117:55294) 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdc359e640 10 monclient(hunting): picked mon.noname-a con 0x7fcdbc02b4c0 addr v1:172.21.15.117:6789/0 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdc359e640 10 monclient(hunting): picked mon.noname-b con 0x7fcdbc0c6610 addr v1:172.21.15.136:6789/0 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdc359e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc0c0760 con 0x7fcdbc02b4c0 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdc359e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc0c3070 con 0x7fcdbc0c6610 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdc359e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdbc0b64b0 con 0x7fcdbc006890 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdc359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdc359e640 10 monclient(hunting): authenticate will time out at 1418.401310s 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdc259c640 1 -- 172.21.15.117:0/2277734204 learned_addr learned my addr 172.21.15.117:0/2277734204 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdb2ffd640 10 client.?.objecter ms_handle_connect 0x7fcdbc006890 2024-03-29T10:44:57.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:57.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690538757 0 0) 0x7fcdbc0b64b0 con 0x7fcdbc006890 2024-03-29T10:44:57.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdb2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:57.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:57.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:57.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdb2ffd640 10 monclient(hunting): my global_id is 4673 2024-03-29T10:44:57.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.139+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda4004eb0 con 0x7fcdbc006890 2024-03-29T10:44:57.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.140+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2805169593 0 0) 0x7fcda4004eb0 con 0x7fcdbc006890 2024-03-29T10:44:57.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.140+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcda4005300 con 0x7fcdbc006890 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.140+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fcdb8001bc0 con 0x7fcdbc006890 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.140+0000 7fcdb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.140+0000 7fcdb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.140+0000 7fcdb2ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:57.124 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:57.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.140+0000 7fcdb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:57.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.140+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1841454622 0 0) 0x7fcda4005300 con 0x7fcdbc006890 2024-03-29T10:44:57.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.140+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 >> v1:172.21.15.136:6789/0 conn(0x7fcdbc0c6610 legacy=0x7fcdbc0b5da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:57.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.143+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 >> v1:172.21.15.117:6789/0 conn(0x7fcdbc02b4c0 legacy=0x7fcdbc15c7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdb2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdb2ffd640 1 monclient: found mon.c 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdb2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc0b6640 con 0x7fcdbc006890 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdb2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdb2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdc359e640 5 monclient: authenticate success, global_id 4673 2024-03-29T10:44:57.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdc359e640 10 monclient: _renew_subs 2024-03-29T10:44:57.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdc359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:57.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdbc0c0bf0 con 0x7fcdbc006890 2024-03-29T10:44:57.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdc359e640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:57.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdc359e640 10 monclient: _renew_subs 2024-03-29T10:44:57.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdc359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:57.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc169030 con 0x7fcdbc006890 2024-03-29T10:44:57.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.144+0000 7fcdc359e640 1 librados: init done 2024-03-29T10:44:57.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.145+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcdb8004a90 con 0x7fcdbc006890 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.145+0000 7fcdb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.145+0000 7fcdc0d99640 4 set_mon_vals no callback set 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.146+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fcdb8001bc0 con 0x7fcdbc006890 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.146+0000 7fcdb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.146+0000 7fcdb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.146+0000 7fcdb2ffd640 10 monclient: dump: 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:44:57.129 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:44:57.130 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:44:57.130 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:44:57.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.147+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fcdb801aa70 con 0x7fcdbc006890 2024-03-29T10:44:57.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.147+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1532302323 0 0) 0x7fcdb8054510 con 0x7fcdbc006890 2024-03-29T10:44:57.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.147+0000 7fcdb2ffd640 10 client.4673.objecter ms_dispatch 0x7fcdc4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:44:57.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.147+0000 7fcdb2ffd640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:44:57.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.147+0000 7fcdb2ffd640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:44:57.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.147+0000 7fcdb2ffd640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T10:44:57.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.149+0000 7fcdc1d9b640 10 monclient: get_auth_request con 0x7fcda406e1c0 auth_method 0 2024-03-29T10:44:57.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.150+0000 7fcdb2ffd640 10 client.4673.objecter ms_handle_connect 0x7fcda406e1c0 2024-03-29T10:44:57.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.150+0000 7fcdc359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:57.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.150+0000 7fcdc359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:57.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.150+0000 7fcdc359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:57.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.150+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda8005080 con 0x7fcdbc006890 2024-03-29T10:44:57.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.158+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcdb80507b0 con 0x7fcdbc006890 2024-03-29T10:44:57.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.158+0000 7fcdb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:57.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.158+0000 7fcdb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:57.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.476+0000 7fcdc359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:57.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.476+0000 7fcdc359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:57.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.476+0000 7fcdc359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T10:44:57.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.476+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcda8005330 con 0x7fcdbc006890 2024-03-29T10:44:57.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.478+0000 7fcdb2ffd640 1 -- 172.21.15.117:0/2277734204 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2172377973) 0x7fcdb80507b0 con 0x7fcdbc006890 2024-03-29T10:44:57.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.478+0000 7fcdb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:57.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.478+0000 7fcdb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:57.462 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:44:57.462 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:44:19.785478+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:19.785478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 10 librados: watch_flush enter 2024-03-29T10:44:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 10 librados: watch_flush exit 2024-03-29T10:44:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T10:44:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 10 client.4673.objecter successfully canceled tick 2024-03-29T10:44:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 >> v1:172.21.15.117:6816/376768171 conn(0x7fcda406e1c0 legacy=0x7fcda4070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 10 monclient: shutdown 2024-03-29T10:44:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 >> v1:172.21.15.117:6790/0 conn(0x7fcdbc006890 legacy=0x7fcdbc0c1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 shutdown_connections 2024-03-29T10:44:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.482+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 >> 172.21.15.117:0/2277734204 conn(0x7fcdbc0058f0 msgr2=0x7fcdbc034f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:57.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.485+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 shutdown_connections 2024-03-29T10:44:57.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.485+0000 7fcdc359e640 1 -- 172.21.15.117:0/2277734204 wait complete. 2024-03-29T10:44:57.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:44:57.486+0000 7fcdc359e640 1 librados: shutdown 2024-03-29T10:44:57.485 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:04.293 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:04.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.460+0000 7fe4c50e6640 10 monclient: get_monmap_and_config 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.460+0000 7fe4c50e6640 10 monclient: build_initial_monmap 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.460+0000 7fe4c50e6640 10 monclient: monmap: 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:45:04.460973+0000 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:45:04.460973+0000 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:45:04.443 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:04.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.461+0000 7fe4c50e6640 1 Processor -- start 2024-03-29T10:45:04.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.461+0000 7fe4c50e6640 1 -- start start 2024-03-29T10:45:04.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.461+0000 7fe4c50e6640 10 monclient: init 2024-03-29T10:45:04.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.462+0000 7fe4c50e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:04.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.462+0000 7fe4c50e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:04.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.462+0000 7fe4c50e6640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c00a9750 addr v1:172.21.15.136:6789/0 2024-03-29T10:45:04.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4bed9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe4c00a9750 0x7fe4c013dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:59928/0 (socket says 172.21.15.117:59928) 2024-03-29T10:45:04.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4be59c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe4c013e1e0 0x7fe4c00ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34859/0 (socket says 172.21.15.117:34859) 2024-03-29T10:45:04.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4bed9d640 1 -- 172.21.15.117:0/4256571955 learned_addr learned my addr 172.21.15.117:0/4256571955 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:45:04.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4c50e6640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c013e1e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:45:04.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4c50e6640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c0071c30 addr v1:172.21.15.117:6789/0 2024-03-29T10:45:04.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4c50e6640 1 -- 172.21.15.117:0/4256571955 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c00ab300 con 0x7fe4c0071c30 2024-03-29T10:45:04.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4c50e6640 1 -- 172.21.15.117:0/4256571955 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c00ac2f0 con 0x7fe4c00a9750 2024-03-29T10:45:04.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4c50e6640 1 -- 172.21.15.117:0/4256571955 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c00ad8d0 con 0x7fe4c013e1e0 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4c50e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4c50e6640 10 monclient(hunting): authenticate will time out at 1425.727361s 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4bdd9b640 1 -- 172.21.15.117:0/4256571955 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746867556 0 0) 0x7fe4c00ad8d0 con 0x7fe4c013e1e0 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4bdd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4bdd9b640 10 monclient(hunting): my global_id is 4676 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.465+0000 7fe4bdd9b640 1 -- 172.21.15.117:0/4256571955 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a8003440 con 0x7fe4c013e1e0 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.466+0000 7fe4bdd9b640 1 -- 172.21.15.117:0/4256571955 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1417046184 0 0) 0x7fe4a8003440 con 0x7fe4c013e1e0 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.466+0000 7fe4bdd9b640 1 -- 172.21.15.117:0/4256571955 >> v1:172.21.15.136:6789/0 conn(0x7fe4c00a9750 legacy=0x7fe4c013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:04.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.466+0000 7fe4bdd9b640 1 -- 172.21.15.117:0/4256571955 >> v1:172.21.15.117:6789/0 conn(0x7fe4c0071c30 legacy=0x7fe4c00ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:04.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:45:04.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 1 monclient: found mon.noname-b 2024-03-29T10:45:04.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:04.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 1 -- 172.21.15.117:0/4256571955 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c00ada60 con 0x7fe4c013e1e0 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: _finish_auth 0 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 1 -- 172.21.15.117:0/4256571955 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe4b8001a10 con 0x7fe4c013e1e0 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: dump: 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:04.452 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:04.453 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:45:04.453 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:45:04.453 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:45:04.453 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:04.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4c50e6640 5 monclient: authenticate success, global_id 4676 2024-03-29T10:45:04.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4c50e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:04.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 1 -- 172.21.15.117:0/4256571955 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b8001be0 con 0x7fe4c013e1e0 2024-03-29T10:45:04.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:04.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 1 -- 172.21.15.117:0/4256571955 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe4b8002030 con 0x7fe4c013e1e0 2024-03-29T10:45:04.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:04.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:04.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4bdd9b640 10 monclient: dump: 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4c50e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.469+0000 7fe4c50e6640 4 set_mon_vals no callback set 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.470+0000 7fe4c50e6640 10 monclient: shutdown 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.470+0000 7fe4c50e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.470+0000 7fe4c50e6640 1 -- 172.21.15.117:0/4256571955 >> v1:172.21.15.117:6790/0 conn(0x7fe4c013e1e0 legacy=0x7fe4c00ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:04.455 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.470+0000 7fe4c50e6640 1 -- 172.21.15.117:0/4256571955 shutdown_connections 2024-03-29T10:45:04.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.470+0000 7fe4c50e6640 1 -- 172.21.15.117:0/4256571955 >> 172.21.15.117:0/4256571955 conn(0x7fe4c006b660 msgr2=0x7fe4c006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:04.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.470+0000 7fe4c50e6640 1 -- 172.21.15.117:0/4256571955 shutdown_connections 2024-03-29T10:45:04.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.471+0000 7fe4c50e6640 1 -- 172.21.15.117:0/4256571955 wait complete. 2024-03-29T10:45:04.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.472+0000 7fe4c50e6640 10 monclient: build_initial_monmap 2024-03-29T10:45:04.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.472+0000 7fe4c50e6640 10 monclient: monmap: 2024-03-29T10:45:04.457 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:45:04.457 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:04.457 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.472+0000 7fe4c50e6640 1 librados: starting msgr at 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.472+0000 7fe4c50e6640 1 librados: starting objecter 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.472+0000 7fe4c50e6640 1 Processor -- start 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.473+0000 7fe4c50e6640 1 -- start start 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.473+0000 7fe4c50e6640 1 librados: setting wanted keys 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.473+0000 7fe4c50e6640 1 librados: calling monclient init 2024-03-29T10:45:04.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.473+0000 7fe4c50e6640 10 monclient: init 2024-03-29T10:45:04.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.473+0000 7fe4c50e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:04.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.473+0000 7fe4c50e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:04.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.476+0000 7fe4c50e6640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c0071c30 addr v1:172.21.15.117:6789/0 2024-03-29T10:45:04.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.476+0000 7fe4bed9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe4c0071c30 0x7fe4c01dbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:47757/0 (socket says 172.21.15.117:47757) 2024-03-29T10:45:04.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.476+0000 7fe4bed9d640 1 -- 172.21.15.117:0/509601112 learned_addr learned my addr 172.21.15.117:0/509601112 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:45:04.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.476+0000 7fe4c50e6640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c00a9750 addr v1:172.21.15.136:6789/0 2024-03-29T10:45:04.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.477+0000 7fe4c50e6640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c013e1e0 addr v1:172.21.15.117:6790/0 2024-03-29T10:45:04.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.477+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c0139940 con 0x7fe4c0071c30 2024-03-29T10:45:04.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.477+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c0140a10 con 0x7fe4c00a9750 2024-03-29T10:45:04.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.477+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c0140cf0 con 0x7fe4c013e1e0 2024-03-29T10:45:04.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.477+0000 7fe4c50e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:04.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.477+0000 7fe4c50e6640 10 monclient(hunting): authenticate will time out at 1425.739883s 2024-03-29T10:45:04.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.480+0000 7fe4a77fe640 10 client.?.objecter ms_handle_connect 0x7fe4c0071c30 2024-03-29T10:45:04.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.480+0000 7fe4a77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.480+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085345104 0 0) 0x7fe4c0139940 con 0x7fe4c0071c30 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.480+0000 7fe4a77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.480+0000 7fe4a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.480+0000 7fe4a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.480+0000 7fe4a77fe640 10 monclient(hunting): my global_id is 4593 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.480+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ac004eb0 con 0x7fe4c0071c30 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 client.?.objecter ms_handle_connect 0x7fe4c013e1e0 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264887286 0 0) 0x7fe4ac004eb0 con 0x7fe4c0071c30 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4c0139940 con 0x7fe4c0071c30 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe4b0001860 con 0x7fe4c0071c30 2024-03-29T10:45:04.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 monclient(hunting): dump: 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813801476 0 0) 0x7fe4c0140cf0 con 0x7fe4c013e1e0 2024-03-29T10:45:04.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:04.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:04.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:04.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 10 monclient(hunting): my global_id is 4679 2024-03-29T10:45:04.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ac004eb0 con 0x7fe4c013e1e0 2024-03-29T10:45:04.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4149537449 0 0) 0x7fe4c0139940 con 0x7fe4c0071c30 2024-03-29T10:45:04.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 >> v1:172.21.15.117:6790/0 conn(0x7fe4c013e1e0 legacy=0x7fe4c0139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.481+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 >> v1:172.21.15.136:6789/0 conn(0x7fe4c00a9750 legacy=0x7fe4c0138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 1 monclient: found mon.a 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c01dc340 con 0x7fe4c0071c30 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 10 client.?.objecter ms_handle_connect 0x7fe4c00a9750 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4a77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4c50e6640 5 monclient: authenticate success, global_id 4593 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4c50e6640 10 monclient: _renew_subs 2024-03-29T10:45:04.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.482+0000 7fe4c50e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:04.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.483+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c01cf140 con 0x7fe4c0071c30 2024-03-29T10:45:04.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.483+0000 7fe4c50e6640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:04.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.483+0000 7fe4c50e6640 10 monclient: _renew_subs 2024-03-29T10:45:04.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.483+0000 7fe4c50e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:04.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.483+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c00b30d0 con 0x7fe4c0071c30 2024-03-29T10:45:04.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.483+0000 7fe4c50e6640 1 librados: init done 2024-03-29T10:45:04.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.484+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b00047e0 con 0x7fe4c0071c30 2024-03-29T10:45:04.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.484+0000 7fe4a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:04.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.484+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7fe4b0004bf0 con 0x7fe4c0071c30 2024-03-29T10:45:04.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.484+0000 7fe4a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:04.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.484+0000 7fe4a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:04.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.484+0000 7fe4a77fe640 10 monclient: dump: 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.484+0000 7fe4bd59a640 4 set_mon_vals no callback set 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.485+0000 7fe4c50e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.485+0000 7fe4c50e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.485+0000 7fe4c50e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:04.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.485+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a0005080 con 0x7fe4c0071c30 2024-03-29T10:45:04.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.485+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7fe4b0004e10 con 0x7fe4c0071c30 2024-03-29T10:45:04.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.486+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1532302323 0 0) 0x7fe4b0084a80 con 0x7fe4c0071c30 2024-03-29T10:45:04.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.486+0000 7fe4a77fe640 10 client.4593.objecter ms_dispatch 0x7fe4c0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:45:04.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.486+0000 7fe4a77fe640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:45:04.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.486+0000 7fe4a77fe640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:45:04.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.486+0000 7fe4a77fe640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-29T10:45:04.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.486+0000 7fe4be59c640 10 monclient: get_auth_request con 0x7fe4ac06dd70 auth_method 0 2024-03-29T10:45:04.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.486+0000 7fe4a77fe640 10 client.4593.objecter ms_handle_connect 0x7fe4ac06dd70 2024-03-29T10:45:04.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.495+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4b0050360 con 0x7fe4c0071c30 2024-03-29T10:45:04.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.495+0000 7fe4a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:04.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.495+0000 7fe4a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:04.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.823+0000 7fe4c50e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:04.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.823+0000 7fe4c50e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:04.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.823+0000 7fe4c50e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:04.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.823+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4a0005330 con 0x7fe4c0071c30 2024-03-29T10:45:04.808 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:45:04.808 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:44:19.785478+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:19.785478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:04.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.824+0000 7fe4a77fe640 1 -- 172.21.15.117:0/509601112 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2172377973) 0x7fe4b0052290 con 0x7fe4c0071c30 2024-03-29T10:45:04.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.824+0000 7fe4a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:04.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.824+0000 7fe4a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:04.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.827+0000 7fe4c50e6640 10 librados: watch_flush enter 2024-03-29T10:45:04.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.827+0000 7fe4c50e6640 10 librados: watch_flush exit 2024-03-29T10:45:04.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.828+0000 7fe4c50e6640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-29T10:45:04.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.828+0000 7fe4c50e6640 10 client.4593.objecter successfully canceled tick 2024-03-29T10:45:04.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.828+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 >> v1:172.21.15.117:6816/376768171 conn(0x7fe4ac06dd70 legacy=0x7fe4ac070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:04.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.828+0000 7fe4c50e6640 10 monclient: shutdown 2024-03-29T10:45:04.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.828+0000 7fe4c50e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:04.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.828+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 >> v1:172.21.15.117:6789/0 conn(0x7fe4c0071c30 legacy=0x7fe4c01dbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:04.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.829+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 shutdown_connections 2024-03-29T10:45:04.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.829+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 >> 172.21.15.117:0/509601112 conn(0x7fe4c006b660 msgr2=0x7fe4c00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:04.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.829+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 shutdown_connections 2024-03-29T10:45:04.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.829+0000 7fe4c50e6640 1 -- 172.21.15.117:0/509601112 wait complete. 2024-03-29T10:45:04.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:04.829+0000 7fe4c50e6640 1 librados: shutdown 2024-03-29T10:45:04.828 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:11.635 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.803+0000 7f8957e20640 10 monclient: get_monmap_and_config 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.803+0000 7f8957e20640 10 monclient: build_initial_monmap 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.803+0000 7f8957e20640 10 monclient: monmap: 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:45:11.804258+0000 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:45:11.804258+0000 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:45:11.786 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-29T10:45:11.787 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T10:45:11.787 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:11.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.804+0000 7f8957e20640 1 Processor -- start 2024-03-29T10:45:11.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 1 -- start start 2024-03-29T10:45:11.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 10 monclient: init 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 10 monclient(hunting): picked mon.noname-a con 0x7f89501457f0 addr v1:172.21.15.117:6789/0 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 10 monclient(hunting): picked mon.noname-b con 0x7f895013f650 addr v1:172.21.15.117:6790/0 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 10 monclient(hunting): picked mon.noname-c con 0x7f89500a9d70 addr v1:172.21.15.136:6789/0 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89500aaef0 con 0x7f89501457f0 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8950145bd0 con 0x7f89500a9d70 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8950145eb0 con 0x7f895013f650 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.805+0000 7f8957e20640 10 monclient(hunting): authenticate will time out at 1433.068022s 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.806+0000 7f8955bbc640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f89501457f0 0x7f895013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:37012/0 (socket says 172.21.15.117:37012) 2024-03-29T10:45:11.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.806+0000 7f8955bbc640 1 -- 172.21.15.117:0/769622711 learned_addr learned my addr 172.21.15.117:0/769622711 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:45:11.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.807+0000 7f8954bba640 1 -- 172.21.15.117:0/769622711 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446796227 0 0) 0x7f89500aaef0 con 0x7f89501457f0 2024-03-29T10:45:11.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.807+0000 7f8954bba640 10 monclient(hunting): _negotiate 2024-03-29T10:45:11.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.807+0000 7f8954bba640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:11.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.807+0000 7f8954bba640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:11.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.807+0000 7f8954bba640 10 monclient(hunting): my global_id is 4596 2024-03-29T10:45:11.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.807+0000 7f8954bba640 1 -- 172.21.15.117:0/769622711 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f893c003520 con 0x7f89501457f0 2024-03-29T10:45:11.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.807+0000 7f8954bba640 1 -- 172.21.15.117:0/769622711 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 535797612 0 0) 0x7f893c003520 con 0x7f89501457f0 2024-03-29T10:45:11.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.807+0000 7f8954bba640 1 -- 172.21.15.117:0/769622711 >> v1:172.21.15.117:6790/0 conn(0x7f895013f650 legacy=0x7f89500b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:11.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 1 -- 172.21.15.117:0/769622711 >> v1:172.21.15.136:6789/0 conn(0x7f89500a9d70 legacy=0x7f895014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: _finish_hunting 0 2024-03-29T10:45:11.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 1 monclient: found mon.noname-a 2024-03-29T10:45:11.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:11.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T10:45:11.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 1 -- 172.21.15.117:0/769622711 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8950146340 con 0x7f89501457f0 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: _finish_auth 0 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: _check_auth_tickets 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 1 -- 172.21.15.117:0/769622711 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8948001a10 con 0x7f89501457f0 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: dump: 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:11.793 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8957e20640 5 monclient: authenticate success, global_id 4596 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8957e20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 1 -- 172.21.15.117:0/769622711 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8948001be0 con 0x7f89501457f0 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 1 -- 172.21.15.117:0/769622711 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f8948002030 con 0x7f89501457f0 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8957e20640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8957e20640 4 set_mon_vals no callback set 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.809+0000 7f8954bba640 10 monclient: dump: 2024-03-29T10:45:11.794 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:45:11.795 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:11.795 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:11.795 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:11.795 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:11.795 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:11.795 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:45:11.795 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:45:11.795 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:45:11.795 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:11.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.810+0000 7f8957e20640 10 monclient: shutdown 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.810+0000 7f8957e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.810+0000 7f8957e20640 1 -- 172.21.15.117:0/769622711 >> v1:172.21.15.117:6789/0 conn(0x7f89501457f0 legacy=0x7f895013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.810+0000 7f8957e20640 1 -- 172.21.15.117:0/769622711 shutdown_connections 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.810+0000 7f8957e20640 1 -- 172.21.15.117:0/769622711 >> 172.21.15.117:0/769622711 conn(0x7f8950066a00 msgr2=0x7f8950066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.811+0000 7f8957e20640 1 -- 172.21.15.117:0/769622711 shutdown_connections 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.811+0000 7f8957e20640 1 -- 172.21.15.117:0/769622711 wait complete. 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.812+0000 7f8957e20640 10 monclient: build_initial_monmap 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.812+0000 7f8957e20640 10 monclient: monmap: 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:11.800 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.812+0000 7f8957e20640 1 librados: starting msgr at 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.812+0000 7f8957e20640 1 librados: starting objecter 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.812+0000 7f8957e20640 1 Processor -- start 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.813+0000 7f8957e20640 1 -- start start 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.813+0000 7f8957e20640 1 librados: setting wanted keys 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.813+0000 7f8957e20640 1 librados: calling monclient init 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.813+0000 7f8957e20640 10 monclient: init 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.813+0000 7f8957e20640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.813+0000 7f8957e20640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f8957e20640 10 monclient(hunting): picked mon.noname-c con 0x7f89500a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T10:45:11.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f8957e20640 10 monclient(hunting): picked mon.noname-a con 0x7f895013f650 addr v1:172.21.15.117:6789/0 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f8957e20640 10 monclient(hunting): picked mon.noname-b con 0x7f89501457f0 addr v1:172.21.15.136:6789/0 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f8957e20640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89500adc70 con 0x7f895013f650 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f8957e20640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89500ad750 con 0x7f89501457f0 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f8957e20640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89500ad900 con 0x7f89500a9d70 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f8957e20640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f8957e20640 10 monclient(hunting): authenticate will time out at 1433.076469s 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f89563bd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f89501457f0 0x7f89500b0ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.117:60838/0 (socket says 172.21.15.117:60838) 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.814+0000 7f89563bd640 1 -- 172.21.15.117:0/2677813284 learned_addr learned my addr 172.21.15.117:0/2677813284 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 10 client.?.objecter ms_handle_connect 0x7f895013f650 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 10 client.?.objecter ms_handle_connect 0x7f89501457f0 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023787507 0 0) 0x7f89500adc70 con 0x7f895013f650 2024-03-29T10:45:11.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 10 monclient(hunting): my global_id is 4599 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.815+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8934004eb0 con 0x7f895013f650 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1745992533 0 0) 0x7f8934004eb0 con 0x7f895013f650 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89500adc70 con 0x7f895013f650 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f894c001860 con 0x7f895013f650 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 10 monclient(hunting): dump: 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:11.803 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097445745 0 0) 0x7f89500ad750 con 0x7f89501457f0 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 10 monclient(hunting): my global_id is 4555 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8934004eb0 con 0x7f89501457f0 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2935508660 0 0) 0x7f89500adc70 con 0x7f895013f650 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.816+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 >> v1:172.21.15.117:6790/0 conn(0x7f89500a9d70 legacy=0x7f8950141e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 >> v1:172.21.15.136:6789/0 conn(0x7f89501457f0 legacy=0x7f89500b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:11.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 1 monclient: found mon.a 2024-03-29T10:45:11.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:11.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:11.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89501d7790 con 0x7f895013f650 2024-03-29T10:45:11.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:11.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:11.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:11.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8957e20640 5 monclient: authenticate success, global_id 4599 2024-03-29T10:45:11.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8957e20640 10 monclient: _renew_subs 2024-03-29T10:45:11.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8957e20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89500ae100 con 0x7f895013f650 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8957e20640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8957e20640 10 monclient: _renew_subs 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8957e20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89500ad3e0 con 0x7f895013f650 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8957e20640 1 librados: init done 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f894c001a30 con 0x7f895013f650 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3808610554 0 0) 0x7f894c003870 con 0x7f895013f650 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f89467fc640 10 monclient: dump: 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T10:45:11.807 INFO:teuthology.misc.health.smithi117.stderr:fsid 539ca054-22ca-4fb7-a22c-88dad95af2f1 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T10:33:36.634637+0000 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.817+0000 7f8947fff640 4 set_mon_vals no callback set 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.818+0000 7f8957e20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.818+0000 7f8957e20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.818+0000 7f8957e20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.818+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8938005080 con 0x7f895013f650 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.820+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2045475259 0 0) 0x7f894c003ab0 con 0x7f895013f650 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.821+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1532302323 0 0) 0x7f894c0837d0 con 0x7f895013f650 2024-03-29T10:45:11.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.821+0000 7f89467fc640 10 client.4599.objecter ms_dispatch 0x7f8950000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:45:11.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.821+0000 7f89467fc640 3 client.4599.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:45:11.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.821+0000 7f89467fc640 3 client.4599.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:45:11.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.821+0000 7f89467fc640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-29T10:45:11.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.821+0000 7f8955bbc640 10 monclient: get_auth_request con 0x7f8934076330 auth_method 0 2024-03-29T10:45:11.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.822+0000 7f89467fc640 10 client.4599.objecter ms_handle_connect 0x7f8934076330 2024-03-29T10:45:11.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.828+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f894c04f0b0 con 0x7f895013f650 2024-03-29T10:45:11.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.828+0000 7f89467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:11.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:11.828+0000 7f89467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:12.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.138+0000 7f8957e20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:12.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.138+0000 7f8957e20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:12.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.138+0000 7f8957e20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T10:45:12.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.138+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8938005330 con 0x7f895013f650 2024-03-29T10:45:12.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.139+0000 7f89467fc640 1 -- 172.21.15.117:0/2677813284 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2172377973) 0x7f894c050fe0 con 0x7f895013f650 2024-03-29T10:45:12.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.139+0000 7f89467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:12.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.139+0000 7f89467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:12.122 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T10:45:12.122 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"539ca054-22ca-4fb7-a22c-88dad95af2f1","created":"2024-03-29T10:34:14.873892+0000","modified":"2024-03-29T10:44:19.785478+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:19.785478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"af857458-18ba-48bb-ba1c-5577c5a7ff12","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"265a899d-aeb3-4ec8-a4df-5ebd03678897","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ccdfb173-0663-4a89-b597-18562ec7c9c9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"89e22417-65aa-443b-8c73-89217c69baaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"31b056b7-6332-4aa0-a4da-21f1f5c7f410","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c98650f7-59e7-4f96-af72-d1369ed68c79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"54aeac70-ee32-4acc-ba23-b1ca8f788b88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f111ef1-b009-4406-abef-7aca27f83384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:12.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 10 librados: watch_flush enter 2024-03-29T10:45:12.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 10 librados: watch_flush exit 2024-03-29T10:45:12.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-29T10:45:12.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 10 client.4599.objecter successfully canceled tick 2024-03-29T10:45:12.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 >> v1:172.21.15.117:6816/376768171 conn(0x7f8934076330 legacy=0x7f89340787f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:12.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 10 monclient: shutdown 2024-03-29T10:45:12.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:12.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 >> v1:172.21.15.117:6789/0 conn(0x7f895013f650 legacy=0x7f89501d7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:12.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 shutdown_connections 2024-03-29T10:45:12.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.142+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 >> 172.21.15.117:0/2677813284 conn(0x7f8950066a00 msgr2=0x7f895013aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:12.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.143+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 shutdown_connections 2024-03-29T10:45:12.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.143+0000 7f8957e20640 1 -- 172.21.15.117:0/2677813284 wait complete. 2024-03-29T10:45:12.126 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T10:45:12.143+0000 7f8957e20640 1 librados: shutdown 2024-03-29T10:45:12.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:12.141 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:45:12.143 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:45:12.143 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:45:12.143 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.173 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:45:12.173 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:45:12.173 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.241 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:45:12.241 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:45:12.241 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.310 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:45:12.311 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:45:12.311 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.381 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:45:12.381 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:45:12.381 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.416 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:45:12.417 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:45:12.417 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.505 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:45:12.505 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:45:12.505 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.582 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:45:12.582 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:45:12.583 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.657 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:45:12.657 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:45:12.658 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:45:12.658 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.683 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:45:12.683 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:45:12.683 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.725 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:45:12.725 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:45:12.725 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:45:12.726 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.737 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:45:12.738 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:45:12.738 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.790 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:45:12.791 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:45:12.791 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:12.801 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:45:12.802 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:45:12.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:45:12.865 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:45:12.865 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:45:15.279 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:45:15.279 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:45:15.366 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:45:15.366 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:45:15.424 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:45:15.424 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:45:15.492 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:45:15.492 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:45:18.281 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:45:18.281 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:45:18.354 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:45:18.354 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:45:18.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:45:18.411 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:45:18.460 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:45:18.462 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/data/mon.a.tgz 2024-03-29T10:45:18.462 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-29T10:45:18.481 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.JkmDTG20T6 2024-03-29T10:45:18.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.JkmDTG20T6 2024-03-29T10:45:18.608 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.JkmDTG20T6 2024-03-29T10:45:18.695 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.JkmDTG20T6 is 211KB 2024-03-29T10:45:18.791 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.JkmDTG20T6 2024-03-29T10:45:18.805 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/data/mon.c.tgz 2024-03-29T10:45:18.806 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-29T10:45:18.860 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.Iw2iCsmVUE 2024-03-29T10:45:18.860 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Iw2iCsmVUE 2024-03-29T10:45:18.961 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.Iw2iCsmVUE 2024-03-29T10:45:19.044 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.Iw2iCsmVUE is 211KB 2024-03-29T10:45:19.109 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.Iw2iCsmVUE 2024-03-29T10:45:19.123 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/data/mon.b.tgz 2024-03-29T10:45:19.123 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-03-29T10:45:19.141 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.CBVIYCG4of 2024-03-29T10:45:19.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.CBVIYCG4of 2024-03-29T10:45:19.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.CBVIYCG4of 2024-03-29T10:45:19.347 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.CBVIYCG4of is 211KB 2024-03-29T10:45:19.440 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.CBVIYCG4of 2024-03-29T10:45:19.455 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:45:19.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:45:19.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:45:19.537 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:45:19.578 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:45:19.578 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:45:19.580 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/remote/smithi117/crash 2024-03-29T10:45:19.581 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:45:19.607 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/remote/smithi136/crash 2024-03-29T10:45:19.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:45:19.645 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:45:19.646 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:45:19.652 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34307.log 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34267.log 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34427.log 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34467.log 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34307.log.gz 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34427.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34267.log.gz 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34467.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:45:19.730 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34387.log 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34427.log.gz 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34467.log.gz 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34387.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr:gzip/var/log/ceph/ceph-osd.7.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34347.log 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34387.log.gz 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34347.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:45:19.731 INFO:teuthology.orchestra.run.smithi136.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34347.log.gz 2024-03-29T10:45:19.732 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34227.log 2024-03-29T10:45:19.732 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34187.log 2024-03-29T10:45:19.732 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:45:19.732 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34227.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34227.log.gz 2024-03-29T10:45:19.732 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34187.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34187.log.gz 2024-03-29T10:45:19.732 INFO:teuthology.orchestra.run.smithi136.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:45:19.744 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:45:19.744 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39348.log 2024-03-29T10:45:19.744 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose --/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.39153.log 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40011.log 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39660.log 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39348.log.gz 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40011.log: /var/log/ceph/ceph-client.admin.35027.log 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39153.log.gz 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40248.log 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40011.log.gz 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35027.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39660.log.gz 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39504.log 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41630.log 2024-03-29T10:45:19.745 INFO:teuthology.orchestra.run.smithi117.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35027.log.gz 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39504.log: gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40248.log.gz -5 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38410.log 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38332.log 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39504.log.gz 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41396.log 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41630.log.gz 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39231.log 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38410.log.gz 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41122.log 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41396.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38332.log.gz 2024-03-29T10:45:19.746 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40888.log 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41396.log.gz 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41240.log 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39231.log.gz 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40888.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41122.log.gz 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41044.log 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41357.log 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40888.log.gz 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41044.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41240.log.gz 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38880.log 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40170.log 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41357.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41044.log.gz 2024-03-29T10:45:19.747 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41005.log 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7%/var/log/ceph/ceph-client.admin.40170.log: -- replaced with /var/log/ceph/ceph-client.admin.41357.log.gz 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41162.log 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38880.log.gz 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41005.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40170.log.gz 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41786.log 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41005.log.gz 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38958.log 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41162.log.gz 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41786.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:45:19.748 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38488.log 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41786.log.gz 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.audit.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38958.log.gz 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39270.log 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40404.log 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39270.log: 93.5%gzip -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41474.log 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38488.log.gz 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40404.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39270.log.gz 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39621.log 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34924.log 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40404.log.gz 2024-03-29T10:45:19.749 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41708.log 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41474.log.gz 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.34924.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39621.log.gz 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38997.log 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34924.log.gz 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41708.log.gz 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39972.log 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:45:19.750 INFO:teuthology.orchestra.run.smithi117.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38997.log.gz 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40482.log 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose 86.7% -- -- replaced with /var/log/ceph/ceph-client.admin.39972.log.gz 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr: /var/log/ceph/ceph-client.admin.39894.log 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41747.log 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40131.log 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40482.log.gz 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41747.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39894.log.gz 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40927.log 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41318.log 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41747.log.gz 2024-03-29T10:45:19.751 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38685.log 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40131.log.gz 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41201.log 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40927.log.gz 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40089.log 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41318.log.gz 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38685.log.gz 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40089.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.41201.log.gz 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41552.log 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40716.log 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40089.log.gz 2024-03-29T10:45:19.752 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41552.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40755.log 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40716.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41552.log.gz 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40755.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40716.log.gz 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40808.log 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39933.log 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40755.log.gz 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40365.log 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38527.log 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40808.log.gz 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39933.log.gz 2024-03-29T10:45:19.753 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39192.log 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40365.log.gz 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41513.log 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39036.log: /var/log/ceph/ceph-client.admin.39192.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38527.log.gz 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40287.log 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41513.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39192.log.gz 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39777.log 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5/var/log/ceph/ceph-client.admin.40287.log: --verbose -- 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41513.log.gz 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr: /var/log/ceph/ceph-client.admin.40849.log 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39777.log: gzip -5 --verbose 87.1% -- /var/log/ceph/ceph-client.admin.38724.log -- replaced with /var/log/ceph/ceph-client.admin.40287.log.gz 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:45:19.754 INFO:teuthology.orchestra.run.smithi117.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39777.log.gz 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40638.log 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38566.log 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40849.log.gz 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40638.log: gzip 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38724.log.gz 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38802.log 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40443.log 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40638.log.gz 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38802.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38566.log.gz 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41279.log 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38802.log.gz 2024-03-29T10:45:19.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39387.log 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40443.log.gz 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40677.log 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40599.log 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41279.log.gz/var/log/ceph/ceph-client.admin.40677.log: 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34633.log 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39387.log.gz 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39543.log 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40677.log.gz 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40599.log: /var/log/ceph/ceph-client.admin.34633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41435.log 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40599.log.gz/var/log/ceph/ceph-client.admin.39543.log: 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28813.log 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34633.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39816.log 2024-03-29T10:45:19.756 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39543.log.gz 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41435.log: /var/log/ceph/ceph.tmp-client.admin.28813.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28813.log.gz 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39582.log 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39816.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41435.log.gz 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39582.log: gzip -5 --verbose 86.4% -- /var/log/ceph/ceph-client.admin.39465.log 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39816.log.gz 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39855.log 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39582.log.gz 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40521.log 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr: 85.7%/var/log/ceph/ceph-client.admin.39855.log: -- replaced with /var/log/ceph/ceph-client.admin.39465.log.gz 2024-03-29T10:45:19.757 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38371.log 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40521.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39855.log.gz 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40209.log 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr: 85.7%/var/log/ceph/ceph-client.admin.38371.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40521.log.gz 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40966.log 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38371.log.gz 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.log: 86.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39426.log 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40209.log.gz 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40966.log: /var/log/ceph/ceph-client.admin.39426.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40966.log.gz 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39426.log.gz 2024-03-29T10:45:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38646.log 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.2.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38449.log 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40050.log 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38646.log: /var/log/ceph/ceph-client.admin.38449.log: 86.4% 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38646.log.gz 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38449.log.gz 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38763.log 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39309.log 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38763.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40050.log.gz 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40560.log 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr: 85.9%/var/log/ceph/ceph-client.admin.39309.log: -- replaced with /var/log/ceph/ceph-client.admin.38763.log.gz 2024-03-29T10:45:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39309.log.gz 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr: /var/log/ceph/ceph-client.admin.38919.log 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr:gzip/var/log/ceph/ceph-client.admin.40560.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41591.log 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0%/var/log/ceph/ceph-client.admin.38919.log: -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39738.log 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41591.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38919.log.gz 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41669.log 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41591.log.gz 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41669.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39738.log.gz 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr: 85.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39075.log 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41669.log.gz 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:45:19.760 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39699.log: /var/log/ceph/ceph-client.admin.39075.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2024-03-29T10:45:19.761 INFO:teuthology.orchestra.run.smithi117.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39075.log.gz 2024-03-29T10:45:19.767 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mon.c.log: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:45:31.012 INFO:teuthology.orchestra.run.smithi117.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:45:31.097 INFO:teuthology.orchestra.run.smithi136.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:45:31.175 INFO:teuthology.orchestra.run.smithi136.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:45:31.212 INFO:teuthology.orchestra.run.smithi136.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:45:35.161 INFO:teuthology.orchestra.run.smithi136.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:45:35.620 INFO:teuthology.orchestra.run.smithi117.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:45:37.119 INFO:teuthology.orchestra.run.smithi117.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:45:37.305 INFO:teuthology.orchestra.run.smithi117.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:46:09.269 INFO:teuthology.orchestra.run.smithi117.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:46:10.097 INFO:teuthology.orchestra.run.smithi117.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:46:10.099 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T10:46:10.099 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m50.436s 2024-03-29T10:46:10.099 INFO:teuthology.orchestra.run.smithi117.stderr:user 2m36.489s 2024-03-29T10:46:10.099 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m5.530s 2024-03-29T10:46:28.481 INFO:teuthology.orchestra.run.smithi136.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T10:46:28.483 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-29T10:46:28.483 INFO:teuthology.orchestra.run.smithi136.stderr:real 1m8.782s 2024-03-29T10:46:28.483 INFO:teuthology.orchestra.run.smithi136.stderr:user 1m54.192s 2024-03-29T10:46:28.483 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m4.182s 2024-03-29T10:46:28.483 INFO:tasks.ceph:Archiving logs... 2024-03-29T10:46:28.483 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/remote/smithi117/log 2024-03-29T10:46:28.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:46:50.703 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/remote/smithi136/log 2024-03-29T10:46:50.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:47:08.212 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:47:08.685 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3c31b484dc3e4f97b5f69a387aaebf5f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:47:08.688 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T10:47:08.697 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T10:47:08.709 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T10:47:08.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:47:08.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:47:08.807 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T10:47:08.815 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T10:47:08.815 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:47:08.818 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:47:08.831 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:47:08.835 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:47:08.836 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:47:08.836 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-29T10:47:08.836 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +441us[ +480us] +/- 66ms 2024-03-29T10:47:08.837 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -198us[ -159us] +/- 66ms 2024-03-29T10:47:08.837 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 -204us[ -165us] +/- 76ms 2024-03-29T10:47:08.837 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:08.840 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:47:08.840 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-29T10:47:08.840 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +463us[ +463us] +/- 66ms 2024-03-29T10:47:08.840 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -137us[ -135us] +/- 66ms 2024-03-29T10:47:08.841 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -190us[ -190us] +/- 76ms 2024-03-29T10:47:08.841 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:08.841 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T10:47:08.850 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T10:47:08.850 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T10:47:08.858 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:47:08.946 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:47:09.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:47:09.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:47:09.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:47:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:47:09.081 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T10:47:09.090 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T10:47:09.102 INFO:teuthology.task.internal:Duration was 1209.489919 seconds 2024-03-29T10:47:09.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T10:47:09.111 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T10:47:09.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:47:09.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:47:09.155 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:47:09.166 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:47:09.449 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T10:47:09.450 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T10:47:09.450 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:47:09.470 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-03-29T10:47:09.471 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:47:09.494 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T10:47:09.494 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:47:09.514 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:47:09.563 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T10:47:09.564 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:47:09.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:47:09.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T10:47:09.710 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T10:47:09.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:47:09.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:47:09.764 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T10:47:09.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:47:09.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:47:09.802 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-03-29T10:47:09.830 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-03-29T10:47:09.865 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:47:09.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:47:09.888 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:47:09.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:47:09.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T10:47:09.931 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T10:47:09.932 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/remote/smithi117 2024-03-29T10:47:09.932 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:47:09.967 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640/remote/smithi136 2024-03-29T10:47:09.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:47:10.001 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T10:47:10.002 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:47:10.004 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:47:10.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T10:47:10.063 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T10:47:10.063 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T10:47:10.074 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T10:47:10.074 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:47:10.077 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:47:10.088 INFO:teuthology.orchestra.run.smithi117.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:47 /home/ubuntu/cephtest 2024-03-29T10:47:10.106 INFO:teuthology.orchestra.run.smithi136.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:47 /home/ubuntu/cephtest 2024-03-29T10:47:10.107 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T10:47:10.117 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T10:47:10.139 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T10:47:10.168 DEBUG:teuthology.nuke:shortname: smithi117 2024-03-29T10:47:10.168 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:47:10.194 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:39.369030', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJuXhtiIzox6RyjMmMKLrraT+wv1+0A0OvY4n+MklX/S7b31OH0r/OKQJhnGds7aodC+WjKO8CpeIt5U8dLrpp0='} 2024-03-29T10:47:10.199 DEBUG:teuthology.nuke:shortname: smithi136 2024-03-29T10:47:10.199 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:47:10.222 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622640', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:39.367052', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBk5LqafpU9UWpW47/yw6pdgu4pgdDLnxkVKd1NImpWtyodgt+PmSbZZ0nRc07yD+cTa1jWNYT9NCp59uKxCtw0='} 2024-03-29T10:47:10.257 INFO:teuthology.orchestra.console.smithi117:Power off 2024-03-29T10:47:10.258 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:47:10.280 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-03-29T10:47:10.280 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:10.299 INFO:teuthology.orchestra.console.smithi136:Power off 2024-03-29T10:47:10.300 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:47:10.313 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-29T10:47:10.327 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-03-29T10:47:10.327 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:10.350 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-29T10:47:14.314 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:14.351 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:14.424 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-29T10:47:14.463 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-29T10:47:18.426 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:18.464 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:18.537 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-03-29T10:47:18.576 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-29T10:47:18.638 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-03-29T10:47:22.581 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:22.691 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-03-29T10:47:22.792 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-03-29T10:47:22.896 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1209.489919424057 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3c31b484dc3e4f97b5f69a387aaebf5f status: fail success: false 2024-03-29T10:47:22.896 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:47:22.982 INFO:teuthology.run:FAIL